date:20191204

aviemzur commented on a change in pull request #6655: [AIRFLOW-6056] Allow 
EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#discussion_r354147011
 
 

 ##
 File path: airflow/contrib/operators/emr_add_steps_operator.py
 ##
 @@ -28,33 +28,58 @@ class EmrAddStepsOperator(BaseOperator):
 
 :param job_flow_id: id of the JobFlow to add steps to. (templated)
 :type job_flow_id: str
+:param job_flow_name: name of the JobFlow to add steps to. Use as an 
alternative to passing
+job_flow_id. will search for id of JobFlow with matching name in one 
of the states in
+param cluster_states. Exactly one cluster like this should exist or 
will fail. (templated)
+:type job_flow_name: str
+:param cluster_states: Acceptable cluster states when searching for 
JobFlow id by job_flow_name.
+(templated)
+:type cluster_states: list
 :param aws_conn_id: aws connection to uses
 :type aws_conn_id: str
 :param steps: boto3 style steps to be added to the jobflow. (templated)
 :type steps: list
+:param do_xcom_push: if True, job_flow_id is pushed to XCom with key 
job_flow_id.
+:type do_xcom_push: bool
 """
-template_fields = ['job_flow_id', 'steps']
+template_fields = ['job_flow_id', 'job_flow_name', 'cluster_states', 
'steps']
 template_ext = ()
 ui_color = '#f9c915'
 
 @apply_defaults
 def __init__(
 self,
-job_flow_id,
-aws_conn_id='aws_default',
+job_flow_id=None,
+job_flow_name=None,
+cluster_states=None,
+aws_conn_id='s3_default',
 
 Review comment:
   It appears that 's3_default' no longer exists[1] and this causes an error 
for users who do not have `s3_default` defined, but I can see how this change 
doesn't relate to this commit. Should I revert this change?
   
   [1] [[AIRFLOW-3682] Use aws_default in EMR related 
operators](https://github.com/apache/airflow/commit/10052e52c2c51062db26f075b2902d14881a0d93)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] aviemzur commented on a change in pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

aviemzur commented on a change in pull request #6655: [AIRFLOW-6056] Allow 
EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#discussion_r354148793
 
 

 ##
 File path: airflow/contrib/operators/emr_add_steps_operator.py
 ##
 @@ -28,33 +28,58 @@ class EmrAddStepsOperator(BaseOperator):
 
 :param job_flow_id: id of the JobFlow to add steps to. (templated)
 :type job_flow_id: str
+:param job_flow_name: name of the JobFlow to add steps to. Use as an 
alternative to passing
+job_flow_id. will search for id of JobFlow with matching name in one 
of the states in
+param cluster_states. Exactly one cluster like this should exist or 
will fail. (templated)
+:type job_flow_name: str
+:param cluster_states: Acceptable cluster states when searching for 
JobFlow id by job_flow_name.
+(templated)
+:type cluster_states: list
 :param aws_conn_id: aws connection to uses
 :type aws_conn_id: str
 :param steps: boto3 style steps to be added to the jobflow. (templated)
 :type steps: list
+:param do_xcom_push: if True, job_flow_id is pushed to XCom with key 
job_flow_id.
+:type do_xcom_push: bool
 """
-template_fields = ['job_flow_id', 'steps']
+template_fields = ['job_flow_id', 'job_flow_name', 'cluster_states', 
'steps']
 template_ext = ()
 ui_color = '#f9c915'
 
 @apply_defaults
 def __init__(
 self,
-job_flow_id,
-aws_conn_id='aws_default',
+job_flow_id=None,
+job_flow_name=None,
+cluster_states=None,
+aws_conn_id='s3_default',
 steps=None,
 *args, **kwargs):
+if kwargs.get('xcom_push') is not None:
+raise AirflowException("'xcom_push' was deprecated, use 
'do_xcom_push' instead")
+if (job_flow_id is None) + (job_flow_name is None) != 1:
 
 Review comment:
   XOR doesn't fulfill the conditional needed here, I can change it to not + 
xor, will this look better?
   ```suggestion
   if not ((job_flow_id is None) ^ (job_flow_name is None)):
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] aviemzur commented on a change in pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

aviemzur commented on a change in pull request #6655: [AIRFLOW-6056] Allow 
EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#discussion_r354148793
 
 

 ##
 File path: airflow/contrib/operators/emr_add_steps_operator.py
 ##
 @@ -28,33 +28,58 @@ class EmrAddStepsOperator(BaseOperator):
 
 :param job_flow_id: id of the JobFlow to add steps to. (templated)
 :type job_flow_id: str
+:param job_flow_name: name of the JobFlow to add steps to. Use as an 
alternative to passing
+job_flow_id. will search for id of JobFlow with matching name in one 
of the states in
+param cluster_states. Exactly one cluster like this should exist or 
will fail. (templated)
+:type job_flow_name: str
+:param cluster_states: Acceptable cluster states when searching for 
JobFlow id by job_flow_name.
+(templated)
+:type cluster_states: list
 :param aws_conn_id: aws connection to uses
 :type aws_conn_id: str
 :param steps: boto3 style steps to be added to the jobflow. (templated)
 :type steps: list
+:param do_xcom_push: if True, job_flow_id is pushed to XCom with key 
job_flow_id.
+:type do_xcom_push: bool
 """
-template_fields = ['job_flow_id', 'steps']
+template_fields = ['job_flow_id', 'job_flow_name', 'cluster_states', 
'steps']
 template_ext = ()
 ui_color = '#f9c915'
 
 @apply_defaults
 def __init__(
 self,
-job_flow_id,
-aws_conn_id='aws_default',
+job_flow_id=None,
+job_flow_name=None,
+cluster_states=None,
+aws_conn_id='s3_default',
 steps=None,
 *args, **kwargs):
+if kwargs.get('xcom_push') is not None:
+raise AirflowException("'xcom_push' was deprecated, use 
'do_xcom_push' instead")
+if (job_flow_id is None) + (job_flow_name is None) != 1:
 
 Review comment:
   XOR doesn't fulfill the conditional I want here, I can change it to not + 
xor, will this look better?
   ```suggestion
   if not ((job_flow_id is None) ^ (job_flow_name is None)):
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] aviemzur commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

aviemzur commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to 
accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#issuecomment-562010507
 
 
   @ashb yeah had a problem with missing packages, now I run the static checks 
locally.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] aviemzur commented on a change in pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

aviemzur commented on a change in pull request #6655: [AIRFLOW-6056] Allow 
EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#discussion_r354147011
 
 

 ##
 File path: airflow/contrib/operators/emr_add_steps_operator.py
 ##
 @@ -28,33 +28,58 @@ class EmrAddStepsOperator(BaseOperator):
 
 :param job_flow_id: id of the JobFlow to add steps to. (templated)
 :type job_flow_id: str
+:param job_flow_name: name of the JobFlow to add steps to. Use as an 
alternative to passing
+job_flow_id. will search for id of JobFlow with matching name in one 
of the states in
+param cluster_states. Exactly one cluster like this should exist or 
will fail. (templated)
+:type job_flow_name: str
+:param cluster_states: Acceptable cluster states when searching for 
JobFlow id by job_flow_name.
+(templated)
+:type cluster_states: list
 :param aws_conn_id: aws connection to uses
 :type aws_conn_id: str
 :param steps: boto3 style steps to be added to the jobflow. (templated)
 :type steps: list
+:param do_xcom_push: if True, job_flow_id is pushed to XCom with key 
job_flow_id.
+:type do_xcom_push: bool
 """
-template_fields = ['job_flow_id', 'steps']
+template_fields = ['job_flow_id', 'job_flow_name', 'cluster_states', 
'steps']
 template_ext = ()
 ui_color = '#f9c915'
 
 @apply_defaults
 def __init__(
 self,
-job_flow_id,
-aws_conn_id='aws_default',
+job_flow_id=None,
+job_flow_name=None,
+cluster_states=None,
+aws_conn_id='s3_default',
 
 Review comment:
   It appears that 's3_default' no longer exists when installing airflow and 
this causes an error.
   But perhaps this is not the commit to fix this issue. Should I revert this 
chang to `aws_conn_id`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on a change in pull request #6652: [AIRFLOW-5548] [AIRFLOW-5550] REST API enhancement - dag info, task …

XD-DENG commented on a change in pull request #6652: [AIRFLOW-5548] 
[AIRFLOW-5550] REST API enhancement - dag info, task …
URL: https://github.com/apache/airflow/pull/6652#discussion_r354141777
 
 

 ##
 File path: airflow/api/common/experimental/__init__.py
 ##
 @@ -45,3 +45,10 @@ def check_and_get_dagrun(dag: DagModel, execution_date: 
datetime) -> DagRun:
  .format(execution_date, dag.dag_id))
 raise DagRunNotFound(error_message)
 return dagrun
+
+
+def check_and_get_dags():
+"""Get the dag ids present in the dagbag"""
+dagbag = DagBag()
 
 Review comment:
   Calling `DagBag()` is a quite expensive action. You may want to get the list 
of dag ids from DB instead.
   
   On the other hand, my personal preference is not to have a new separate 
function if what you're doing inside is very straightforward & not being used 
at multiple locations (here this `check_and_get_dags()` function is only used 
at one location).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6696: [AIRFLOW-6128] Simplify plugins_manager and webserver plugin code

codecov-io edited a comment on issue #6696: [AIRFLOW-6128] Simplify 
plugins_manager and webserver plugin code
URL: https://github.com/apache/airflow/pull/6696#issuecomment-560183548
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6696?src=pr=h1) 
Report
   > Merging 
[#6696](https://codecov.io/gh/apache/airflow/pull/6696?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/7f040fce5200bf2c4ca22c6a22c91bcdc2405101?src=pr=desc)
 will **decrease** coverage by `0.08%`.
   > The diff coverage is `89.08%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6696/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6696?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6696  +/-   ##
   ==
   - Coverage   83.92%   83.84%   -0.09% 
   ==
 Files 668  668  
 Lines   3772237747  +25 
   ==
   - Hits3166031649  -11 
   - Misses   6062 6098  +36
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6696?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6696/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `76.77% <ø> (ø)` | :arrow_up: |
   | 
[airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/6696/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=)
 | `91.5% <100%> (ø)` | :arrow_up: |
   | 
[airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/6696/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=)
 | `91.09% <84.21%> (+1.88%)` | :arrow_up: |
   | 
[airflow/www/app.py](https://codecov.io/gh/apache/airflow/pull/6696/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBwLnB5)
 | `94.11% <92.78%> (-1.65%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6696/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6696/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `85.33% <0%> (-4%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6696/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `91.52% <0%> (-1.7%)` | :arrow_down: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6696/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `89.83% <0%> (-1.7%)` | :arrow_down: |
   | 
[airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/6696/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5)
 | `92.95% <0%> (-1.41%)` | :arrow_down: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6696/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `89.96% <0%> (-1.22%)` | :arrow_down: |
   | ... and [1 
more](https://codecov.io/gh/apache/airflow/pull/6696/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6696?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6696?src=pr=footer). 
Last update 
[7f040fc...0eea374](https://codecov.io/gh/apache/airflow/pull/6696?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on issue #6731: [AIRFLOW-XXX] Increase log level for clarity

XD-DENG commented on issue #6731: [AIRFLOW-XXX] Increase log level for clarity
URL: https://github.com/apache/airflow/pull/6731#issuecomment-561953664
 
 
   Hi @robinedwards 
   
   - This is a code change, so JIRA ticket is required.
   - You can always change logging level in your configuration, so I don't find 
this change necessary.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] KevinYang21 commented on issue #5079: [AIRFLOW-4285] Update task dependency context defination and usage

KevinYang21 commented on issue #5079: [AIRFLOW-4285] Update task dependency 
context defination and usage
URL: https://github.com/apache/airflow/pull/5079#issuecomment-561905372
 
 
   Thanks @ashb for clarifying! I'll do a quick deep dive later today on this. 
From what you described, my understanding is that basically our problem came 
from [how we sort the queued tasks when 
triggering](https://github.com/apache/airflow/blob/master/airflow/executors/base_executor.py#L159-L162).
 So when we trigger tasks we always let those upstream tasks to take over the 
open slots and leave the downstream tasks in the queue_tasks. I think the 
one-line fix you had is needed logically but I'm not 100% sure if that is 
solving the root cause for us. In fact the behavior strikes me as expected, 
since we don'g really set priority on those tasks.
   
   All above is my hypothesis, I'll validate when I got a charger for my 
laptop--docker burns out battery too fast...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman opened a new pull request #6732: fixes scheduler queue bug

dimberman opened a new pull request #6732: fixes scheduler queue bug
URL: https://github.com/apache/airflow/pull/6732

Make sure you have checked _all_ steps below.

### Jira

- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6175
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).

### Description

- [ ] Here are some details about my PR, including screenshots of any UI
changes:

There is a bug caused by scheduler_jobs refactor which leads to task failure
and scheduler locking.

Essentially when a there is an overflow of tasks going into the scheduler,
the tasks are set back to scheduled, but are not removed from the executor's
queued_tasks queue.

This means that the executor will attempt to run tasks that are in the
scheduled state, but those tasks will fail dependency checks. Eventually the
queue is filled with scheduled tasks, and the scheduler can no longer run.
### Tests

- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:

### Commits

- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Documentation

- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6175) Executor changes queued tasks to scheduled without removing from task queue

2019-12-04 Thread Daniel Imberman (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Imberman updated AIRFLOW-6175:
-
Description: 
There is a bug caused by scheduler_jobs refactor which leads to task failure 
and scheduler locking.

Essentially when a there is an overflow of tasks going into the scheduler, the 
tasks are set back to scheduled, but are not removed from the executor's 
queued_tasks queue.

This means that the executor will attempt to run tasks that are in the 
scheduled state, but those tasks will fail dependency checks. Eventually the 
queue is filled with scheduled tasks, and the scheduler can no longer run.

> Executor changes queued tasks to scheduled without removing from task queue
> ---
>
> Key: AIRFLOW-6175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6175
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.6
>Reporter: Daniel Imberman
>Priority: Blocker
>
> There is a bug caused by scheduler_jobs refactor which leads to task failure 
> and scheduler locking.
> Essentially when a there is an overflow of tasks going into the scheduler, 
> the tasks are set back to scheduled, but are not removed from the executor's 
> queued_tasks queue.
> This means that the executor will attempt to run tasks that are in the 
> scheduled state, but those tasks will fail dependency checks. Eventually the 
> queue is filled with scheduled tasks, and the scheduler can no longer run.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-6175) Executor changes queued tasks to scheduled without removing from task queue

2019-12-04 Thread Daniel Imberman (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Imberman updated AIRFLOW-6175:
-
Fix Version/s: 1.10.6

> Executor changes queued tasks to scheduled without removing from task queue
> ---
>
> Key: AIRFLOW-6175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6175
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.6
>Reporter: Daniel Imberman
>Priority: Blocker
> Fix For: 1.10.6
>
>
> There is a bug caused by scheduler_jobs refactor which leads to task failure 
> and scheduler locking.
> Essentially when a there is an overflow of tasks going into the scheduler, 
> the tasks are set back to scheduled, but are not removed from the executor's 
> queued_tasks queue.
> This means that the executor will attempt to run tasks that are in the 
> scheduled state, but those tasks will fail dependency checks. Eventually the 
> queue is filled with scheduled tasks, and the scheduler can no longer run.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6176) Form submits change url from https to http when redirectoung

2019-12-04 Thread Ronald Fenner (Jira)

Ronald Fenner created AIRFLOW-6176:
--

 Summary: Form submits change url from https to http when 
redirectoung
 Key: AIRFLOW-6176
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6176
 Project: Apache Airflow
  Issue Type: Bug
  Components: ui
Affects Versions: 1.10.6
 Environment: amazon linux 2
Reporter: Ronald Fenner


We server the airflow web server behind an ALB in AWS that terminates SSL for 
us. when submitting forms such as adding a variable when clicking save it 
changes the url to http from https and we get a bad request.

It seems that the initial post call by the form works as the variable gets 
created but then when we get redirected back to the list is when the url drops 
the https.

 

I've seen this behavior as well then creating a pool. The pool gets created but 
we get routed to an http page afterwards.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results

codecov-io edited a comment on issue #6075: [AIRFLOW-5266] Allow 
aws_athena_hook to get all query results
URL: https://github.com/apache/airflow/pull/6075#issuecomment-561876309
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=h1) 
Report
   > Merging 
[#6075](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/b355fd65ac44a65ee94c30d3797509c370e64986?src=pr=desc)
 will **decrease** coverage by `0.31%`.
   > The diff coverage is `16.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6075/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6075  +/-   ##
   ==
   - Coverage   83.91%   83.59%   -0.32% 
   ==
 Files 668  668  
 Lines   3772037732  +12 
   ==
   - Hits3165231543 -109 
   - Misses   6068 6189 +121
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/providers/amazon/aws/hooks/athena.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9hdGhlbmEucHk=)
 | `60.81% <16.66%> (-6.38%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `77.85% <0%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | ... and [15 
more](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=footer). 
Last update 
[b355fd6...aaed88c](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results

2019-12-04 Thread ASF subversion and git services (Jira)

codecov-io commented on issue #6075: [AIRFLOW-5266] Allow aws_athena_hook to 
get all query results
URL: https://github.com/apache/airflow/pull/6075#issuecomment-561876309
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=h1) 
Report
   > Merging 
[#6075](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/b355fd65ac44a65ee94c30d3797509c370e64986?src=pr=desc)
 will **decrease** coverage by `0.86%`.
   > The diff coverage is `16.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6075/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6075  +/-   ##
   ==
   - Coverage   83.91%   83.04%   -0.87% 
   ==
 Files 668  668  
 Lines   3772037732  +12 
   ==
   - Hits3165231336 -316 
   - Misses   6068 6396 +328
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/providers/amazon/aws/hooks/athena.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9hdGhlbmEucHk=)
 | `60.81% <16.66%> (-6.38%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `44.96% <0%> (-32.89%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | ... and [15 
more](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=footer). 
Last update 
[b355fd6...aaed88c](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6175) Executor changes queued tasks to scheduled without removing from task queue

2019-12-04 Thread Daniel Imberman (Jira)

Daniel Imberman created AIRFLOW-6175:


 Summary: Executor changes queued tasks to scheduled without 
removing from task queue
 Key: AIRFLOW-6175
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6175
 Project: Apache Airflow
  Issue Type: Bug
  Components: scheduler
Affects Versions: 1.10.6
Reporter: Daniel Imberman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6174) Airflow Databricks Hook requires host in extras

2019-12-04 Thread Costas Piliotis (Jira)

Costas Piliotis created AIRFLOW-6174:


 Summary: Airflow Databricks Hook requires host in extras
 Key: AIRFLOW-6174
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6174
 Project: Apache Airflow
  Issue Type: Bug
  Components: hooks
Affects Versions: 1.10.6
Reporter: Costas Piliotis


databricks_hook was changed to require host in the extras in the connection.   

 

[https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/databricks_hook.py#L154]

 

Maybe first check and see if host is provided in the connection before 
searching extras?    



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6165) Housekeeping for utils.dates.date_range(), and add tests for it



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988238#comment-16988238
 ] 

ASF subversion and git services commented on AIRFLOW-6165:
--

Commit 7f040fce5200bf2c4ca22c6a22c91bcdc2405101 in airflow's branch 
refs/heads/master from Xiaodong
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7f040fc ]

[AIRFLOW-6165] Housekeep utils.dates.date_range & add tests (#6720)



> Housekeeping for utils.dates.date_range(), and add tests for it
> ---
>
> Key: AIRFLOW-6165
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6165
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: utils
>Affects Versions: 1.10.6
>Reporter: Xiaodong Deng
>Assignee: Xiaodong Deng
>Priority: Minor
> Fix For: 2.0.0
>
>
> There are some issues in *utils.dates.date_range()*, like missing docstring, 
> missing proper handle on wrong input, etc. And it doesn't come with enough 
> test coverage.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6165) Housekeeping for utils.dates.date_range(), and add tests for it

2019-12-04 Thread Kaxil Naik (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-6165.
-
Resolution: Fixed

> Housekeeping for utils.dates.date_range(), and add tests for it
> ---
>
> Key: AIRFLOW-6165
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6165
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: utils
>Affects Versions: 1.10.6
>Reporter: Xiaodong Deng
>Assignee: Xiaodong Deng
>Priority: Minor
> Fix For: 2.0.0
>
>
> There are some issues in *utils.dates.date_range()*, like missing docstring, 
> missing proper handle on wrong input, etc. And it doesn't come with enough 
> test coverage.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6165) Housekeeping for utils.dates.date_range(), and add tests for it



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988237#comment-16988237
 ] 

ASF GitHub Bot commented on AIRFLOW-6165:
-

kaxil commented on pull request #6720: [AIRFLOW-6165] Housekeep 
utils.dates.date_range & add tests
URL: https://github.com/apache/airflow/pull/6720
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Housekeeping for utils.dates.date_range(), and add tests for it
> ---
>
> Key: AIRFLOW-6165
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6165
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: utils
>Affects Versions: 1.10.6
>Reporter: Xiaodong Deng
>Assignee: Xiaodong Deng
>Priority: Minor
> Fix For: 2.0.0
>
>
> There are some issues in *utils.dates.date_range()*, like missing docstring, 
> missing proper handle on wrong input, etc. And it doesn't come with enough 
> test coverage.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil merged pull request #6720: [AIRFLOW-6165] Housekeep utils.dates.date_range & add tests

kaxil merged pull request #6720: [AIRFLOW-6165] Housekeep 
utils.dates.date_range & add tests
URL: https://github.com/apache/airflow/pull/6720
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6140) Add missing types and pylint compliance for core classes

2019-12-04 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988197#comment-16988197
 ] 

ASF GitHub Bot commented on AIRFLOW-6140:
-

potiuk commented on pull request #6702: [AIRFLOW-6140] Add missing types for 
some core classes.
URL: https://github.com/apache/airflow/pull/6702
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add missing types and pylint compliance for core classes
> 
>
> Key: AIRFLOW-6140
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6140
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Assignee: Jarek Potiuk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6140) Add missing types and pylint compliance for core classes



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988198#comment-16988198
 ] 

ASF subversion and git services commented on AIRFLOW-6140:
--

Commit b355fd65ac44a65ee94c30d3797509c370e64986 in airflow's branch 
refs/heads/master from Jarek Potiuk
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b355fd6 ]

[AIRFLOW-6140] Add missing types for some core classes (#6702)



> Add missing types and pylint compliance for core classes
> 
>
> Key: AIRFLOW-6140
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6140
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Assignee: Jarek Potiuk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6140) Add missing types and pylint compliance for core classes



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6140.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Add missing types and pylint compliance for core classes
> 
>
> Key: AIRFLOW-6140
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6140
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Assignee: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #6702: [AIRFLOW-6140] Add missing types for some core classes.

potiuk merged pull request #6702: [AIRFLOW-6140] Add missing types for some 
core classes.
URL: https://github.com/apache/airflow/pull/6702
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #6728: [AIRFLOW-XXX] Add Changelog for 1.10.6

potiuk merged pull request #6728: [AIRFLOW-XXX] Add Changelog for 1.10.6
URL: https://github.com/apache/airflow/pull/6728
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file

codecov-io edited a comment on issue #6230: [AIRFLOW-5413] Allow K8S worker pod 
to be configured from JSON/YAML file
URL: https://github.com/apache/airflow/pull/6230#issuecomment-537654940
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=h1) 
Report
   > Merging 
[#6230](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/e5ff4a17fe73ceed5fd4369e95a2409238333e68?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `92.56%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6230/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #6230  +/-   ##
   =
   - Coverage83.9%   83.9%   -0.01% 
   =
 Files 668 668  
 Lines   37687   37764  +77 
   =
   + Hits31622   31684  +62 
   - Misses   60656080  +15
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/kubernetes/secret.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3NlY3JldC5weQ==)
 | `93.61% <100%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5)
 | `96.42% <100%> (ø)` | :arrow_up: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `98.76% <100%> (+0.04%)` | :arrow_up: |
   | 
[airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=)
 | `92.96% <91.39%> (-1.7%)` | :arrow_down: |
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `59.24% <93.33%> (-0.81%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `77.85% <0%> (ø)` | :arrow_up: |
   | ... and [13 
more](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=footer). 
Last update 
[e5ff4a1...06e942e](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file

codecov-io edited a comment on issue #6230: [AIRFLOW-5413] Allow K8S worker pod 
to be configured from JSON/YAML file
URL: https://github.com/apache/airflow/pull/6230#issuecomment-537654940
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=h1) 
Report
   > Merging 
[#6230](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/e5ff4a17fe73ceed5fd4369e95a2409238333e68?src=pr=desc)
 will **decrease** coverage by `0.55%`.
   > The diff coverage is `92.56%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6230/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6230  +/-   ##
   ==
   - Coverage83.9%   83.35%   -0.56% 
   ==
 Files 668  668  
 Lines   3768737764  +77 
   ==
   - Hits3162231477 -145 
   - Misses   6065 6287 +222
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/kubernetes/secret.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3NlY3JldC5weQ==)
 | `93.61% <100%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5)
 | `96.42% <100%> (ø)` | :arrow_up: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `98.76% <100%> (+0.04%)` | :arrow_up: |
   | 
[airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=)
 | `92.96% <91.39%> (-1.7%)` | :arrow_down: |
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `59.24% <93.33%> (-0.81%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `44.96% <0%> (-32.89%)` | :arrow_down: |
   | ... and [13 
more](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=footer). 
Last update 
[e5ff4a1...06e942e](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6731: [AIRFLOW-XXX] Increase log level for clarity

codecov-io edited a comment on issue #6731: [AIRFLOW-XXX] Increase log level 
for clarity
URL: https://github.com/apache/airflow/pull/6731#issuecomment-561785567
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=h1) 
Report
   > Merging 
[#6731](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/e5ff4a17fe73ceed5fd4369e95a2409238333e68?src=pr=desc)
 will **decrease** coverage by `0.32%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6731/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6731  +/-   ##
   ==
   - Coverage83.9%   83.58%   -0.33% 
   ==
 Files 668  668  
 Lines   3768737687  
   ==
   - Hits3162231499 -123 
   - Misses   6065 6188 +123
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `52.13% <ø> (-0.39%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `77.85% <0%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | ... and [12 
more](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=footer). 
Last update 
[e5ff4a1...e924262](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

ashb commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to 
accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#issuecomment-561785572
 
 
   You also need to fix the failing static checks. You should be able to run 
those locally.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6731: [AIRFLOW-XXX] Increase log level for clarity

codecov-io commented on issue #6731: [AIRFLOW-XXX] Increase log level for 
clarity
URL: https://github.com/apache/airflow/pull/6731#issuecomment-561785567
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=h1) 
Report
   > Merging 
[#6731](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/e5ff4a17fe73ceed5fd4369e95a2409238333e68?src=pr=desc)
 will **decrease** coverage by `0.85%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6731/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6731  +/-   ##
   ==
   - Coverage83.9%   83.04%   -0.86% 
   ==
 Files 668  668  
 Lines   3768737687  
   ==
   - Hits3162231298 -324 
   - Misses   6065 6389 +324
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `52.13% <ø> (-0.39%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `44.96% <0%> (-32.89%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | ... and [12 
more](https://codecov.io/gh/apache/airflow/pull/6731/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=footer). 
Last update 
[e5ff4a1...e924262](https://codecov.io/gh/apache/airflow/pull/6731?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

ashb commented on a change in pull request #6655: [AIRFLOW-6056] Allow 
EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#discussion_r353918334
 
 

 ##
 File path: airflow/contrib/operators/emr_add_steps_operator.py
 ##
 @@ -28,33 +28,58 @@ class EmrAddStepsOperator(BaseOperator):
 
 :param job_flow_id: id of the JobFlow to add steps to. (templated)
 :type job_flow_id: str
+:param job_flow_name: name of the JobFlow to add steps to. Use as an 
alternative to passing
+job_flow_id. will search for id of JobFlow with matching name in one 
of the states in
+param cluster_states. Exactly one cluster like this should exist or 
will fail. (templated)
+:type job_flow_name: str
+:param cluster_states: Acceptable cluster states when searching for 
JobFlow id by job_flow_name.
+(templated)
+:type cluster_states: list
 :param aws_conn_id: aws connection to uses
 :type aws_conn_id: str
 :param steps: boto3 style steps to be added to the jobflow. (templated)
 :type steps: list
+:param do_xcom_push: if True, job_flow_id is pushed to XCom with key 
job_flow_id.
+:type do_xcom_push: bool
 """
-template_fields = ['job_flow_id', 'steps']
+template_fields = ['job_flow_id', 'job_flow_name', 'cluster_states', 
'steps']
 template_ext = ()
 ui_color = '#f9c915'
 
 @apply_defaults
 def __init__(
 self,
-job_flow_id,
-aws_conn_id='aws_default',
+job_flow_id=None,
+job_flow_name=None,
+cluster_states=None,
+aws_conn_id='s3_default',
 steps=None,
 *args, **kwargs):
+if kwargs.get('xcom_push') is not None:
+raise AirflowException("'xcom_push' was deprecated, use 
'do_xcom_push' instead")
+if (job_flow_id is None) + (job_flow_name is None) != 1:
 
 Review comment:
   This is a slightly odd way of expressing this. Either use logcial XOR:
   
   ```suggestion
   if (job_flow_id is None) ^ (job_flow_name is None):
   ```
   
   Or do it as
   
   ```suggestion
   if job_flow_id is None or job_flow_name is None or (job_flow_name 
and job_flow_id):
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

ashb commented on a change in pull request #6655: [AIRFLOW-6056] Allow 
EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#discussion_r353916418
 
 

 ##
 File path: airflow/contrib/operators/emr_add_steps_operator.py
 ##
 @@ -28,33 +28,58 @@ class EmrAddStepsOperator(BaseOperator):
 
 :param job_flow_id: id of the JobFlow to add steps to. (templated)
 :type job_flow_id: str
+:param job_flow_name: name of the JobFlow to add steps to. Use as an 
alternative to passing
+job_flow_id. will search for id of JobFlow with matching name in one 
of the states in
+param cluster_states. Exactly one cluster like this should exist or 
will fail. (templated)
+:type job_flow_name: str
+:param cluster_states: Acceptable cluster states when searching for 
JobFlow id by job_flow_name.
+(templated)
+:type cluster_states: list
 :param aws_conn_id: aws connection to uses
 :type aws_conn_id: str
 :param steps: boto3 style steps to be added to the jobflow. (templated)
 :type steps: list
+:param do_xcom_push: if True, job_flow_id is pushed to XCom with key 
job_flow_id.
+:type do_xcom_push: bool
 """
-template_fields = ['job_flow_id', 'steps']
+template_fields = ['job_flow_id', 'job_flow_name', 'cluster_states', 
'steps']
 template_ext = ()
 ui_color = '#f9c915'
 
 @apply_defaults
 def __init__(
 self,
-job_flow_id,
-aws_conn_id='aws_default',
+job_flow_id=None,
+job_flow_name=None,
+cluster_states=None,
+aws_conn_id='s3_default',
 
 Review comment:
   You've changed the default connection ID. You shouldn't.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] robinedwards opened a new pull request #6731: [AIRFLOW-XXX] Increase log level for clarity

robinedwards opened a new pull request #6731: [AIRFLOW-XXX] Increase log level 
for clarity
URL: https://github.com/apache/airflow/pull/6731
 
 
   This case prevents a DAG from being scheduled by the scheduler so should be 
an error to make troubleshooting easier.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6660: [AIRFLOW-6065] Add Stackdriver Task Handler

codecov-io edited a comment on issue #6660: [AIRFLOW-6065] Add Stackdriver Task 
Handler
URL: https://github.com/apache/airflow/pull/6660#issuecomment-561766454
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=h1) 
Report
   > Merging 
[#6660](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/0a0d09714bc499d2c2dafc60a68fa7fea7dcab75?src=pr=desc)
 will **decrease** coverage by `0.29%`.
   > The diff coverage is `92.79%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6660/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master#6660 +/-   ##
   =
   - Coverage   83.88%   83.59%   -0.3% 
   =
 Files 668  669  +1 
 Lines   3768437798+114 
   =
   - Hits3161231598 -14 
   - Misses   6072 6200+128
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=)
 | `56.6% <16.66%> (-5.1%)` | :arrow_down: |
   | 
[airflow/utils/log/stackdriver\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvc3RhY2tkcml2ZXJfdGFza19oYW5kbGVyLnB5)
 | `97.14% <97.14%> (ø)` | |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.2% <0%> (-20.52%)` | :arrow_down: |
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `89.13% <0%> (-3.63%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `93.22% <0%> (ø)` | :arrow_up: |
   | ... and [9 
more](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=footer). 
Last update 
[0a0d097...de62509](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6660: [AIRFLOW-6065] Add Stackdriver Task Handler

codecov-io commented on issue #6660: [AIRFLOW-6065] Add Stackdriver Task Handler
URL: https://github.com/apache/airflow/pull/6660#issuecomment-561766454
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=h1) 
Report
   > Merging 
[#6660](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/0a0d09714bc499d2c2dafc60a68fa7fea7dcab75?src=pr=desc)
 will **decrease** coverage by `0.35%`.
   > The diff coverage is `92.79%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6660/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6660  +/-   ##
   ==
   - Coverage   83.88%   83.52%   -0.36% 
   ==
 Files 668  669   +1 
 Lines   3768437798 +114 
   ==
   - Hits3161231572  -40 
   - Misses   6072 6226 +154
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=)
 | `56.6% <16.66%> (-5.1%)` | :arrow_down: |
   | 
[airflow/utils/log/stackdriver\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvc3RhY2tkcml2ZXJfdGFza19oYW5kbGVyLnB5)
 | `97.14% <97.14%> (ø)` | |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.2% <0%> (-20.52%)` | :arrow_down: |
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `89.13% <0%> (-3.63%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `91.52% <0%> (-1.7%)` | :arrow_down: |
   | ... and [9 
more](https://codecov.io/gh/apache/airflow/pull/6660/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=footer). 
Last update 
[0a0d097...de62509](https://codecov.io/gh/apache/airflow/pull/6660?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6095) Task stats endpoint is slow and returns a large payload



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988047#comment-16988047
 ] 

ASF GitHub Bot commented on AIRFLOW-6095:
-

robinedwards commented on pull request #6730: [AIRFLOW-6095] Filter dags 
returned by task_stats
URL: https://github.com/apache/airflow/pull/6730
 
 
   Add dag_ids parameter to task_stats so can filter by a set of dag_ids
   present on the dags view. This is intended to speed up the response time
   and reduce the size of the payload when running a large number of dags.
   
   (cherry picked from commit 2b11f55b406d38b59c3062ad75826ad6c1b1fdb9) this 
back ports the functionality from master in to v1-10-test
   
   ### Jira
   
   - [x] My PR addresses the following
 - https://issues.apache.org/jira/browse/AIRFLOW-6095
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Add dag_ids parameter to task_stats so can filter by a set of dag_ids
   present on the dags view. This is intended to speed up the response time
   and reduce the size of the payload when running a large number of dags.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   N/A
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Task stats endpoint is slow and returns a large payload
> ---
>
> Key: AIRFLOW-6095
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6095
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: api, ui
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Robin Edwards
>Assignee: Robin Edwards
>Priority: Minor
> Fix For: 1.10.7
>
>
> The task_stats endpoint returns all dags by default. This can result in an 
> extremely large payload ~ 8mb and slow response time when you have a lot of 
> dags (In this case 1500+).
> The accompanying pull request adds a dag_ids get parameter to the task_stats 
> end point which is populated by the dags present on the page.
> If this is accepted I intend to provide the equivalent change to the 
> dag_stats and last_dagruns endpoints



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] robinedwards opened a new pull request #6730: [AIRFLOW-6095] Filter dags returned by task_stats

robinedwards opened a new pull request #6730: [AIRFLOW-6095] Filter dags
returned by task_stats
URL: https://github.com/apache/airflow/pull/6730

Add dag_ids parameter to task_stats so can filter by a set of dag_ids
present on the dags view. This is intended to speed up the response time
and reduce the size of the payload when running a large number of dags.

(cherry picked from commit 2b11f55b406d38b59c3062ad75826ad6c1b1fdb9) this
back ports the functionality from master in to v1-10-test

### Jira

- [x] My PR addresses the following
- https://issues.apache.org/jira/browse/AIRFLOW-6095
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).

### Description

- [x] Here are some details about my PR, including screenshots of any UI
changes:

### Tests

- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:

### Commits

- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Documentation

- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release

N/A

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6729: [AIRFLOW-6172] BigQuery - Move BigQuery hook Pandas data frame system…

codecov-io edited a comment on issue #6729: [AIRFLOW-6172] BigQuery - Move 
BigQuery hook Pandas data frame system…
URL: https://github.com/apache/airflow/pull/6729#issuecomment-561743784
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=h1) 
Report
   > Merging 
[#6729](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/e5ff4a17fe73ceed5fd4369e95a2409238333e68?src=pr=desc)
 will **decrease** coverage by `0.32%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6729/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6729  +/-   ##
   ==
   - Coverage83.9%   83.58%   -0.33% 
   ==
 Files 668  668  
 Lines   3768737687  
   ==
   - Hits3162231500 -122 
   - Misses   6065 6187 +122
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.2% <0%> (-20.52%)` | :arrow_down: |
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `89.13% <0%> (-3.63%)` | :arrow_down: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `91.18% <0%> (-0.31%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `52.32% <0%> (-0.2%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=footer). 
Last update 
[e5ff4a1...2162ee0](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6729: [AIRFLOW-6172] BigQuery - Move BigQuery hook Pandas data frame system…

codecov-io commented on issue #6729: [AIRFLOW-6172] BigQuery - Move BigQuery 
hook Pandas data frame system…
URL: https://github.com/apache/airflow/pull/6729#issuecomment-561743784
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=h1) 
Report
   > Merging 
[#6729](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/e5ff4a17fe73ceed5fd4369e95a2409238333e68?src=pr=desc)
 will **decrease** coverage by `0.32%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6729/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6729  +/-   ##
   ==
   - Coverage83.9%   83.58%   -0.33% 
   ==
 Files 668  668  
 Lines   3768737687  
   ==
   - Hits3162231500 -122 
   - Misses   6065 6187 +122
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.2% <0%> (-20.52%)` | :arrow_down: |
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `89.13% <0%> (-3.63%)` | :arrow_down: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `91.18% <0%> (-0.31%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6729/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `52.32% <0%> (-0.2%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=footer). 
Last update 
[e5ff4a1...2162ee0](https://codecov.io/gh/apache/airflow/pull/6729?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-6173) Remote logging handler does not close at the end of run

2019-12-04 Thread Ali Soleimani Dehdivan (Jira)

Ali Soleimani Dehdivan created AIRFLOW-6173:
---

 Summary: Remote logging handler does not close at the end of run
 Key: AIRFLOW-6173
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6173
 Project: Apache Airflow
  Issue Type: Bug
  Components: cli, gcp
Affects Versions: 1.10.6, 1.10.5
Reporter: Ali Soleimani Dehdivan


Hi, We tried to enable remote logging to GCS bucket but we could not manage to 
run it successfully after digging in the code I found that if I close 
GCSTaskHandler manually in cli.py line  552 after _run it works fine. I added 
these lines 

for h in reversed(ti.log.handlers[:]):
     h.close()

I'm not sure if it is a bug or not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ashb commented on issue #5079: [AIRFLOW-4285] Update task dependency context defination and usage

ashb commented on issue #5079: [AIRFLOW-4285] Update task dependency context 
defination and usage
URL: https://github.com/apache/airflow/pull/5079#issuecomment-561716214
 
 
   @KevinYang21 Confirmed, it still happens, and seems to be an issue when the 
tasks "overflow" the executor.
   
   For example, set:
   
   - AIRFLOW__CORE__PARALLELISM=5
   - ```
 dag = DAG('example_dag',
max_active_runs=50,
concurrency=500,```
   
   We've got a one-line fix for it too:
   
   
https://github.com/astronomer/airflow/commit/8d0ee05976db021d4f3185d5ca55ae818fc1f7a0
   
   (It took about 5 person-days to track down! Always the way with one line 
fixes.)
   
   Once we've got unit tests added too we'll open a PR. Unfortunately this bug 
also affects 1.10.6 as we cherry-picked this change :(


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6172) BigQuery - Move BigQuery hook Pandas data frame system tests to separate file



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987956#comment-16987956
 ] 

ASF GitHub Bot commented on AIRFLOW-6172:
-

TobKed commented on pull request #6729: [AIRFLOW-6172] BigQuery - Move BigQuery 
hook Pandas data frame system…
URL: https://github.com/apache/airflow/pull/6729
 
 
   … tests to separate file
   
   This is done due to the fact that these tests require authorization to 
BigQuery
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-6172
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> BigQuery - Move BigQuery hook Pandas data frame system tests to separate file
> -
>
> Key: AIRFLOW-6172
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6172
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: gcp, hooks
>Affects Versions: 1.10.6
>Reporter: Tobiasz Kedzierski
>Assignee: Tobiasz Kedzierski
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #6702: [AIRFLOW-6140] Add missing types for some core classes.

codecov-io edited a comment on issue #6702: [AIRFLOW-6140] Add missing types 
for some core classes.
URL: https://github.com/apache/airflow/pull/6702#issuecomment-560257785
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=h1) 
Report
   > Merging 
[#6702](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/e5ff4a17fe73ceed5fd4369e95a2409238333e68?src=pr=desc)
 will **decrease** coverage by `0.31%`.
   > The diff coverage is `87.36%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6702/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6702  +/-   ##
   ==
   - Coverage83.9%   83.59%   -0.32% 
   ==
 Files 668  668  
 Lines   3768737720  +33 
   ==
   - Hits3162231532  -90 
   - Misses   6065 6188 +123
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `89.13% <ø> (-3.63%)` | :arrow_down: |
   | 
[airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==)
 | `91.22% <100%> (+0.27%)` | :arrow_up: |
   | 
[airflow/macros/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9tYWNyb3MvX19pbml0X18ucHk=)
 | `82.35% <100%> (+1.1%)` | :arrow_up: |
   | 
[airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=)
 | `60.86% <65.51%> (-0.84%)` | :arrow_down: |
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `96.05% <95.74%> (-0.25%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | ... and [3 
more](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=footer). 
Last update 
[e5ff4a1...3669652](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #6683: [AIRFLOW-6019] Add flushing in execute method for BigQueryCursor

mik-laj commented on issue #6683: [AIRFLOW-6019] Add flushing in execute method 
for BigQueryCursor
URL: https://github.com/apache/airflow/pull/6683#issuecomment-561703213
 
 
   Can you add tests to prevent regression?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6702: [AIRFLOW-6140] Add missing types for some core classes.

codecov-io edited a comment on issue #6702: [AIRFLOW-6140] Add missing types 
for some core classes.
URL: https://github.com/apache/airflow/pull/6702#issuecomment-560257785
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=h1) 
Report
   > Merging 
[#6702](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/e5ff4a17fe73ceed5fd4369e95a2409238333e68?src=pr=desc)
 will **decrease** coverage by `0.31%`.
   > The diff coverage is `87.36%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6702/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6702  +/-   ##
   ==
   - Coverage83.9%   83.59%   -0.32% 
   ==
 Files 668  668  
 Lines   3768737720  +33 
   ==
   - Hits3162231532  -90 
   - Misses   6065 6188 +123
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `89.13% <ø> (-3.63%)` | :arrow_down: |
   | 
[airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==)
 | `91.22% <100%> (+0.27%)` | :arrow_up: |
   | 
[airflow/macros/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9tYWNyb3MvX19pbml0X18ucHk=)
 | `82.35% <100%> (+1.1%)` | :arrow_up: |
   | 
[airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=)
 | `60.86% <65.51%> (-0.84%)` | :arrow_down: |
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `96.05% <95.74%> (-0.25%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | ... and [3 
more](https://codecov.io/gh/apache/airflow/pull/6702/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=footer). 
Last update 
[e5ff4a1...3669652](https://codecov.io/gh/apache/airflow/pull/6702?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] TobKed opened a new pull request #6729: [AIRFLOW-6172] BigQuery - Move BigQuery hook Pandas data frame system…

TobKed opened a new pull request #6729: [AIRFLOW-6172] BigQuery - Move BigQuery 
hook Pandas data frame system…
URL: https://github.com/apache/airflow/pull/6729
 
 
   … tests to separate file
   
   This is done due to the fact that these tests require authorization to 
BigQuery
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-6172
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-6172) BigQuery - Move BigQuery hook Pandas data frame system tests to separate file

2019-12-04 Thread Tobiasz Kedzierski (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tobiasz Kedzierski updated AIRFLOW-6172:

Summary: BigQuery - Move BigQuery hook Pandas data frame system tests to 
separate file  (was: BigQuery - Move BigQuery Pandas data frame system tests to 
separate file)

> BigQuery - Move BigQuery hook Pandas data frame system tests to separate file
> -
>
> Key: AIRFLOW-6172
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6172
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: gcp, hooks
>Affects Versions: 1.10.6
>Reporter: Tobiasz Kedzierski
>Assignee: Tobiasz Kedzierski
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-6172) BigQuery - Move BigQuery Pandas data frame system tests to separate file

2019-12-04 Thread Tobiasz Kedzierski (Jira)

Tobiasz Kedzierski created AIRFLOW-6172:
---

 Summary: BigQuery - Move BigQuery Pandas data frame system tests 
to separate file
 Key: AIRFLOW-6172
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6172
 Project: Apache Airflow
  Issue Type: Improvement
  Components: gcp, hooks
Affects Versions: 1.10.6
Reporter: Tobiasz Kedzierski
Assignee: Tobiasz Kedzierski






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] aviemzur edited a comment on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

aviemzur edited a comment on issue #6655: [AIRFLOW-6056] Allow 
EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#issuecomment-561699359
 
 
   @ashb Added commit with the changes you requested + lint fixes [Changes 
after CR 
2](https://github.com/apache/airflow/pull/6655/commits/abae66a9c4a8968aea7718052370c2571122e2ee)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] aviemzur commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

aviemzur commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to 
accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#issuecomment-561699359
 
 
   @ashb Added commit with the changes you requested [Changes after CR 
2](https://github.com/apache/airflow/pull/6655/commits/abae66a9c4a8968aea7718052370c2571122e2ee)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6472: [AIRFLOW-6058] Running tests with pytest

codecov-io edited a comment on issue #6472: [AIRFLOW-6058] Running tests with 
pytest
URL: https://github.com/apache/airflow/pull/6472#issuecomment-558580149
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6472?src=pr=h1) 
Report
   > Merging 
[#6472](https://codecov.io/gh/apache/airflow/pull/6472?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/e5ff4a17fe73ceed5fd4369e95a2409238333e68?src=pr=desc)
 will **increase** coverage by `0.73%`.
   > The diff coverage is `57.14%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6472/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6472?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6472  +/-   ##
   ==
   + Coverage83.9%   84.64%   +0.73% 
   ==
 Files 668  669   +1 
 Lines   3768737703  +16 
   ==
   + Hits3162231912 +290 
   + Misses   6065 5791 -274
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6472?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6472/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `75.93% <ø> (-0.85%)` | :arrow_down: |
   | 
[airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6472/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=)
 | `89.58% <57.14%> (-3.6%)` | :arrow_down: |
   | 
[airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6472/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==)
 | `56% <0%> (-44%)` | :arrow_down: |
   | 
[...low/ti\_deps/deps/exec\_date\_after\_start\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/6472/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvZXhlY19kYXRlX2FmdGVyX3N0YXJ0X2RhdGVfZGVwLnB5)
 | `80% <0%> (-10%)` | :arrow_down: |
   | 
[airflow/executors/executor\_loader.py](https://codecov.io/gh/apache/airflow/pull/6472/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvZXhlY3V0b3JfbG9hZGVyLnB5)
 | `59.09% <0%> (-6.82%)` | :arrow_down: |
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6472/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `88.07% <0%> (-3.67%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6472/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `89.83% <0%> (-3.39%)` | :arrow_down: |
   | 
[airflow/utils/log/s3\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6472/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvczNfdGFza19oYW5kbGVyLnB5)
 | `98.5% <0%> (-1.5%)` | :arrow_down: |
   | 
[airflow/executors/base\_executor.py](https://codecov.io/gh/apache/airflow/pull/6472/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvYmFzZV9leGVjdXRvci5weQ==)
 | `96.05% <0%> (-1.32%)` | :arrow_down: |
   | ... and [21 
more](https://codecov.io/gh/apache/airflow/pull/6472/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6472?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6472?src=pr=footer). 
Last update 
[e5ff4a1...b0514d5](https://codecov.io/gh/apache/airflow/pull/6472?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil opened a new pull request #6728: [AIRFLOW-XXX] Add Changelog for 1.10.6

kaxil opened a new pull request #6728: [AIRFLOW-XXX] Add Changelog for 1.10.6
URL: https://github.com/apache/airflow/pull/6728
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
   
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   Add Changelog for 1.10.6
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dougblack closed pull request #6721: Pod aliases simple

dougblack closed pull request #6721: Pod aliases simple
URL: https://github.com/apache/airflow/pull/6721
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5645) In quickstart, precise how airflow commands can be monitored from web UI



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987883#comment-16987883
 ] 

ASF GitHub Bot commented on AIRFLOW-5645:
-

stale[bot] commented on pull request #6316: [AIRFLOW-5645] Precise webserver 
status monitoring possibilities in quickstart
URL: https://github.com/apache/airflow/pull/6316
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> In quickstart, precise how airflow commands can be monitored from web UI
> 
>
> Key: AIRFLOW-5645
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5645
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 1.10.5
>Reporter: Thomas Pilewicz
>Assignee: Thomas Pilewicz
>Priority: Minor
>
> The quickstart suggests to run two airflow commands from the CLI to trigger 
> some DAG runs and task instances : airflow tasks run, and airflow dags 
> backfill. It states that the status of the jobs will be displayed to the user 
> but doesn't mention that the webserver can be used to monitor them.
> I suggest developing this statement to tell the user that he will have 
> feedback from the CLI, but can also see the status of his tasks in the web 
> UI, which is a significant feature of Airflow.
> I'll send a PR with this formulation if that's OK



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] stale[bot] closed pull request #6316: [AIRFLOW-5645] Precise webserver status monitoring possibilities in quickstart

stale[bot] closed pull request #6316: [AIRFLOW-5645] Precise webserver status 
monitoring possibilities in quickstart
URL: https://github.com/apache/airflow/pull/6316
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] baolsen commented on issue #6686: [AIRFLOW-6072] aws_hook: Outbound http proxy setting and other enhancements

baolsen commented on issue #6686: [AIRFLOW-6072] aws_hook: Outbound http proxy 
setting and other enhancements
URL: https://github.com/apache/airflow/pull/6686#issuecomment-561641382
 
 
   Thanks @potiuk that makes sense. My build passed now, so likely a flaky test 
as you said :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6686: [AIRFLOW-6072] aws_hook: Outbound http proxy setting and other enhancements

codecov-io edited a comment on issue #6686: [AIRFLOW-6072] aws_hook: Outbound 
http proxy setting and other enhancements
URL: https://github.com/apache/airflow/pull/6686#issuecomment-559765152
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6686?src=pr=h1) 
Report
   > Merging 
[#6686](https://codecov.io/gh/apache/airflow/pull/6686?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/9fed459441ab94015f79b7b8ed6ee9f42a89cca8?src=pr=desc)
 will **increase** coverage by `0.13%`.
   > The diff coverage is `65.9%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6686/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6686?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6686  +/-   ##
   ==
   + Coverage   83.83%   83.97%   +0.13% 
   ==
 Files 668  671   +3 
 Lines   3756941003+3434 
   ==
   + Hits3149734431+2934 
   - Misses   6072 6572 +500
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6686?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/providers/amazon/aws/operators/datasync.py](https://codecov.io/gh/apache/airflow/pull/6686/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvZGF0YXN5bmMucHk=)
 | `25.44% <0%> (-0.46%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/aws\_hook.py](https://codecov.io/gh/apache/airflow/pull/6686/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F3c19ob29rLnB5)
 | `78.15% <69.04%> (+1.13%)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6686/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6686/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6686/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6686/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6686/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.2% <0%> (-20.52%)` | :arrow_down: |
   | 
[airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6686/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==)
 | `85.29% <0%> (-14.71%)` | :arrow_down: |
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6686/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `89.13% <0%> (-3.63%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6686/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `57.26% <0%> (-0.73%)` | :arrow_down: |
   | ... and [31 
more](https://codecov.io/gh/apache/airflow/pull/6686/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6686?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6686?src=pr=footer). 
Last update 
[9fed459...9319d4d](https://codecov.io/gh/apache/airflow/pull/6686?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on issue #6721: Pod aliases simple

XD-DENG commented on issue #6721: Pod aliases simple
URL: https://github.com/apache/airflow/pull/6721#issuecomment-561637272
 
 
   Hi @dougblack , a gentle reminder for following the suggestions/requirements 
described in the PR template.
   
   On the other hand, as I suggested in 
https://github.com/apache/airflow/pull/6673, please create the PR only after 
your changes are somehow in shape. This can help shorten the CI build queue. If 
you want to make use of Travis CI, you can do it in your own fork as well 
(actually the builds will be triggered faster there based on my experience)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] yuqian90 commented on issue #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker

2019-12-04 Thread ASF subversion and git services (Jira)

yuqian90 commented on issue #6633: [AIRFLOW-2279] Clear tasks across DAGs if 
marked by ExternalTaskMarker
URL: https://github.com/apache/airflow/pull/6633#issuecomment-561636948
 
 
   @kaxil @OmerJog the failing static check has been fixed and all tests 
passed. Pls let me know what you think about the PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6169) Avoid unnecessary int-to-float conversion in utils.dates.scale_time_units



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987843#comment-16987843
 ] 

ASF subversion and git services commented on AIRFLOW-6169:
--

Commit e5ff4a17fe73ceed5fd4369e95a2409238333e68 in airflow's branch 
refs/heads/master from Xiaodong
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e5ff4a1 ]

[AIRFLOW-6169] Avoid unnecessary int-to-float conversion (#6724)



> Avoid unnecessary int-to-float conversion in utils.dates.scale_time_units
> -
>
> Key: AIRFLOW-6169
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6169
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: utils
>Affects Versions: 1.10.6
>Reporter: Xiaodong Deng
>Assignee: Xiaodong Deng
>Priority: Trivial
> Fix For: 2.0.0
>
>
> The "x * 0.1" is doing int-to-float type conversion, which is not necessary 
> anymore in Py3 context.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6169) Avoid unnecessary int-to-float conversion in utils.dates.scale_time_units

2019-12-04 Thread Xiaodong Deng (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaodong Deng resolved AIRFLOW-6169.

Resolution: Resolved

> Avoid unnecessary int-to-float conversion in utils.dates.scale_time_units
> -
>
> Key: AIRFLOW-6169
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6169
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: utils
>Affects Versions: 1.10.6
>Reporter: Xiaodong Deng
>Assignee: Xiaodong Deng
>Priority: Trivial
> Fix For: 2.0.0
>
>
> The "x * 0.1" is doing int-to-float type conversion, which is not necessary 
> anymore in Py3 context.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6169) Avoid unnecessary int-to-float conversion in utils.dates.scale_time_units



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987842#comment-16987842
 ] 

ASF GitHub Bot commented on AIRFLOW-6169:
-

XD-DENG commented on pull request #6724: [AIRFLOW-6169] Avoid unnecessary 
int-to-float conversion
URL: https://github.com/apache/airflow/pull/6724
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Avoid unnecessary int-to-float conversion in utils.dates.scale_time_units
> -
>
> Key: AIRFLOW-6169
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6169
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: utils
>Affects Versions: 1.10.6
>Reporter: Xiaodong Deng
>Assignee: Xiaodong Deng
>Priority: Trivial
> Fix For: 2.0.0
>
>
> The "x * 0.1" is doing int-to-float type conversion, which is not necessary 
> anymore in Py3 context.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] XD-DENG merged pull request #6724: [AIRFLOW-6169] Avoid unnecessary int-to-float conversion

XD-DENG merged pull request #6724: [AIRFLOW-6169] Avoid unnecessary 
int-to-float conversion
URL: https://github.com/apache/airflow/pull/6724
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #6686: [AIRFLOW-6072] aws_hook: Outbound http proxy setting and other enhancements

potiuk commented on issue #6686: [AIRFLOW-6072] aws_hook: Outbound http proxy 
setting and other enhancements
URL: https://github.com/apache/airflow/pull/6686#issuecomment-561623921
 
 
   It looked like it was a flaky test. I restarted just this job that failed. 
Stabilising such flaky tests is sometimes tricky.
   
   However. we are just about to merge a change to use pytests rather than 
nosetests as test execution engine (hopefully today @nuclearpinguin  ). So 
this will happen less often we hope.
   
   This will enable us to mark some tests as flaky and have pytest retry such 
flaky tests in case they fail (and we will continue fixing  such flaky tests in 
parallel).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-5934) TRAINING ISSUE! Add extra CC: to the emails sent by Airflow



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-5934:
--
Description: 
{quote}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)

{color:#ff}This is a test issue that is used in Contributor's documentation 
to show an example issue for the first time contributors.{color}

{color:#ff}Please do not merge the change even if you implement it!{color}

{color:#ff}The example workflow that uses this JIRA issue is described in 
{color}

{color:#ff}[https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example]{color}

{color:#ff}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!){color}
{quote}
We need to have a way to add extra CC: to  the emails sent by Airflow.
 # You can see the information about how to configure email here email: 
[https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration]
 # Here is the class you should modify:  
[https://github.com/apache/airflow/blob/master/airflow/utils/email.py] 
 # Tests for email class are in: 
[https://github.com/apache/airflow/blob/master/tests/utils/test_email.py] 
 # Run the tests in Local Virtualenv: 
[https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst]
 # Run the tests in Breeze: 
[https://github.com/apache/airflow/blob/master/BREEZE.rst]
 # Make a DRAFT PR as described in 
[https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions]
 # Run airflow webserver and scheduler and configure it to send emails and 
connect to a real SMTP server and test if the email is sent with the CC:'d 
account

  was:
{quote}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)

{color:#ff}This is a test issue that is used in Contributor's documentation 
to show an example issue for the first time contributors.{color}

{color:#ff}Please do not merge the change even if you implement it!{color}

{color:#ff}The example workflow that uses this JIRA issue is described in 
[Contrbution Workflow 
Example|[https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example]]{color}

{color:#ff}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!){color}
{quote}
We need to have a way to add extra CC: to  the emails sent by Airflow.
 # You can see the information about how to configure email here email: 
[https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration]
 # Here is the class you should modify:  
[https://github.com/apache/airflow/blob/master/airflow/utils/email.py] 
 # Tests for email class are in: 
[https://github.com/apache/airflow/blob/master/tests/utils/test_email.py] 
 # Run the tests in Local Virtualenv: 
[https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst]
 # Run the tests in Breeze: 
[https://github.com/apache/airflow/blob/master/BREEZE.rst]
 # Make a DRAFT PR as described in 
[https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions]
 # Run airflow webserver and scheduler and configure it to send emails and 
connect to a real SMTP server and test if the email is sent with the CC:'d 
account


> TRAINING ISSUE! Add extra CC: to the emails sent by Airflow
> ---
>
> Key: AIRFLOW-5934
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5934
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>  Labels: training-issue-
>
> {quote}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)
> {color:#ff}This is a test issue that is used in Contributor's 
> documentation to show an example issue for the first time contributors.{color}
> {color:#ff}Please do not merge the change even if you implement it!{color}
> {color:#ff}The example workflow that uses this JIRA issue is described in 
> {color}
> {color:#ff}[https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example]{color}
> {color:#ff}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!){color}
> {quote}
> We need to have a way to add extra CC: to  the emails sent by Airflow.
>  # You can see the information about how to configure email here email: 
>

[jira] [Updated] (AIRFLOW-5934) TRAINING ISSUE! Add extra CC: to the emails sent by Airflow



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-5934:
--
Description: 
{quote}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)

{color:#ff}This is a test issue that is used in Contributor's documentation 
to show an example issue for the first time contributors.{color}

{color:#ff}Please do not merge the change even if you implement it!{color}

{color:#ff}The example workflow that uses this JIRA issue is described in 
[Contrbution Workflow 
Example|[https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example]]{color}

{color:#ff}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!){color}
{quote}
We need to have a way to add extra CC: to  the emails sent by Airflow.
 # You can see the information about how to configure email here email: 
[https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration]
 # Here is the class you should modify:  
[https://github.com/apache/airflow/blob/master/airflow/utils/email.py] 
 # Tests for email class are in: 
[https://github.com/apache/airflow/blob/master/tests/utils/test_email.py] 
 # Run the tests in Local Virtualenv: 
[https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst]
 # Run the tests in Breeze: 
[https://github.com/apache/airflow/blob/master/BREEZE.rst]
 # Make a DRAFT PR as described in 
[https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions]
 # Run airflow webserver and scheduler and configure it to send emails and 
connect to a real SMTP server and test if the email is sent with the CC:'d 
account

  was:
{quote}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)

{color:#ff}This is a test issue that is used in Contributor's documentation 
to show an example issue for the first time contributors.{color}

{color:#ff}Please do not merge the change even if you implement it!{color}

{color:#ff}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!){color}
{quote}
 

We need to have a way to add extra CC: to  the emails sent by Airflow.
 # You can see the information about how to configure email here email: 
[https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration]
 # Here is the class you should modify:  
[https://github.com/apache/airflow/blob/master/airflow/utils/email.py] 
 # Tests for email class are in: 
[https://github.com/apache/airflow/blob/master/tests/utils/test_email.py] 
 # Run the tests in Local Virtualenv: 
[https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst]
 # Run the tests in Breeze: 
[https://github.com/apache/airflow/blob/master/BREEZE.rst]
 # Make a DRAFT PR as described in 
[https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions]
 # Run airflow webserver and scheduler and configure it to send emails and 
connect to a real SMTP server and test if the email is sent with the CC:'d 
account


> TRAINING ISSUE! Add extra CC: to the emails sent by Airflow
> ---
>
> Key: AIRFLOW-5934
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5934
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>  Labels: training-issue-
>
> {quote}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)
> {color:#ff}This is a test issue that is used in Contributor's 
> documentation to show an example issue for the first time contributors.{color}
> {color:#ff}Please do not merge the change even if you implement it!{color}
> {color:#ff}The example workflow that uses this JIRA issue is described in 
> [Contrbution Workflow 
> Example|[https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example]]{color}
> {color:#ff}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!){color}
> {quote}
> We need to have a way to add extra CC: to  the emails sent by Airflow.
>  # You can see the information about how to configure email here email: 
> [https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration]
>  # Here is the class you should modify:  
> [https://github.com/apache/airflow/blob/master/airflow/utils/email.py]

[jira] [Updated] (AIRFLOW-5934) TRAINING ISSUE! Add extra CC: to the emails sent by Airflow



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-5934:
--
Description: 
{quote}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)

{color:#ff}This is a test issue that is used in Contributor's documentation 
to show an example issue for the first time contributors.{color}

{color:#ff}Please do not merge the change even if you implement it!{color}

{color:#ff}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!){color}
{quote}
 

We need to have a way to add extra CC: to  the emails sent by Airflow.
 # You can see the information about how to configure email here email: 
[https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration]
 # Here is the class you should modify:  
[https://github.com/apache/airflow/blob/master/airflow/utils/email.py] 
 # Tests for email class are in: 
[https://github.com/apache/airflow/blob/master/tests/utils/test_email.py] 
 # Run the tests in Local Virtualenv: 
[https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst]
 # Run the tests in Breeze: 
[https://github.com/apache/airflow/blob/master/BREEZE.rst]
 # Make a DRAFT PR as described in 
[https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions]
 # Run airflow webserver and scheduler and configure it to send emails and 
connect to a real SMTP server and test if the email is sent with the CC:'d 
account

  was:
{quote}{color:#FF}NOTE! THIS IS A TEST issue that is used in Contributor's 
documentation to show an example issue for the first time contributors. Please 
do not implement it :). !{color}
{quote}
We need to have a way to add extra CC: to  the emails sent by Airflow.
 # You can see the information about how to configure email here email: 
[https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration]
 # Here is the class you should modify:  
[https://github.com/apache/airflow/blob/master/airflow/utils/email.py] 
 # Tests for email class are in: 
[https://github.com/apache/airflow/blob/master/tests/utils/test_email.py] 
 # Run the tests in Local Virtualenv: 
[https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst]
 # Run the tests in Breeze: 
[https://github.com/apache/airflow/blob/master/BREEZE.rst]
 # Make a DRAFT PR as described in 
[https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions]
 # Run airflow webserver and scheduler and configure it to send emails and 
connect to a real SMTP server and test if the email is sent with the CC:'d 
account


> TRAINING ISSUE! Add extra CC: to the emails sent by Airflow
> ---
>
> Key: AIRFLOW-5934
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5934
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>  Labels: training-issue-
>
> {quote}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)
> {color:#ff}This is a test issue that is used in Contributor's 
> documentation to show an example issue for the first time contributors.{color}
> {color:#ff}Please do not merge the change even if you implement it!{color}
> {color:#ff}(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!)(!){color}
> {quote}
>  
> We need to have a way to add extra CC: to  the emails sent by Airflow.
>  # You can see the information about how to configure email here email: 
> [https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration]
>  # Here is the class you should modify:  
> [https://github.com/apache/airflow/blob/master/airflow/utils/email.py] 
>  # Tests for email class are in: 
> [https://github.com/apache/airflow/blob/master/tests/utils/test_email.py] 
>  # Run the tests in Local Virtualenv: 
> [https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst]
>  # Run the tests in Breeze: 
> [https://github.com/apache/airflow/blob/master/BREEZE.rst]
>  # Make a DRAFT PR as described in 
> [https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions]
>  # Run airflow webserver and scheduler and configure it to send emails and 
> connect to a real SMTP server and test if the email is sent with the CC:'d 
> account



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #6585: More GSOD improvements

2019-12-04 Thread ASF subversion and git services (Jira)

potiuk merged pull request #6585: More GSOD improvements
URL: https://github.com/apache/airflow/pull/6585
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6007) Doc check checks provider/ instead of providers/ packages



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987770#comment-16987770
 ] 

ASF subversion and git services commented on AIRFLOW-6007:
--

Commit 37a1bd80f82b37a6278597c67d6a2c855e7d2ecb in airflow's branch 
refs/heads/master from Jarek Potiuk
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=37a1bd8 ]

[AIRFLOW-6007] Check providers instead of provider package (#6599)



> Doc check checks provider/ instead of providers/ packages
> -
>
> Key: AIRFLOW-6007
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6007
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on a change in pull request #6702: [AIRFLOW-6140] Add missing types for some core classes.

potiuk commented on a change in pull request #6702: [AIRFLOW-6140] Add missing 
types for some core classes.
URL: https://github.com/apache/airflow/pull/6702#discussion_r353707769
 
 

 ##
 File path: .travis.yml
 ##
 @@ -24,7 +24,7 @@ env:
 - MOUNT_LOCAL_SOURCES="false"
 - AIRFLOW_MOUNT_HOST_VOLUMES_FOR_STATIC_CHECKS="true"
 - FORCE_ANSWER_TO_QUESTIONS="yes"
-- AIRFLOW_CI_SILENT="true"
+- AIRFLOW_CI_SILENT="false"
 
 Review comment:
   True. It was for debugging of a problem. Will revert.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-6007) Doc check checks provider/ instead of providers/ packages



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-6007.
---
Fix Version/s: 2.0.0
   Resolution: Fixed

> Doc check checks provider/ instead of providers/ packages
> -
>
> Key: AIRFLOW-6007
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6007
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6007) Doc check checks provider/ instead of providers/ packages



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987769#comment-16987769
 ] 

ASF GitHub Bot commented on AIRFLOW-6007:
-

potiuk commented on pull request #6599: [AIRFLOW-6007] Check providers instead 
of provider package.
URL: https://github.com/apache/airflow/pull/6599
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Doc check checks provider/ instead of providers/ packages
> -
>
> Key: AIRFLOW-6007
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6007
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #6726: [AIRFLOW-6170] BranchPythonOperator does not do XCom push of returned value

codecov-io edited a comment on issue #6726: [AIRFLOW-6170] BranchPythonOperator 
does not do XCom push of returned value
URL: https://github.com/apache/airflow/pull/6726#issuecomment-561617155
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6726?src=pr=h1) 
Report
   > Merging 
[#6726](https://codecov.io/gh/apache/airflow/pull/6726?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/0a0d09714bc499d2c2dafc60a68fa7fea7dcab75?src=pr=desc)
 will **decrease** coverage by `0.3%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6726/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6726?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6726  +/-   ##
   ==
   - Coverage   83.88%   83.58%   -0.31% 
   ==
 Files 668  668  
 Lines   3768437688   +4 
   ==
   - Hits3161231501 -111 
   - Misses   6072 6187 +115
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6726?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/python\_operator.py](https://codecov.io/gh/apache/airflow/pull/6726/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHl0aG9uX29wZXJhdG9yLnB5)
 | `95.8% <100%> (+0.02%)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6726/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6726/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6726/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6726/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6726/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `78.2% <0%> (-20.52%)` | :arrow_down: |
   | 
[airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6726/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5)
 | `89.13% <0%> (-3.63%)` | :arrow_down: |
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6726/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `89.33% <0%> (-0.67%)` | :arrow_down: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6726/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `91.18% <0%> (-0.31%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6726/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `52.32% <0%> (-0.2%)` | :arrow_down: |
   | ... and [3 
more](https://codecov.io/gh/apache/airflow/pull/6726/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6726?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6726?src=pr=footer). 
Last update 
[0a0d097...31cda1a](https://codecov.io/gh/apache/airflow/pull/6726?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6725: [AIRFLOW-6157] Support for multiple executors

potiuk commented on a change in pull request #6725: [AIRFLOW-6157] Support for 
multiple executors
URL: https://github.com/apache/airflow/pull/6725#discussion_r353706504
 
 

 ##
 File path: airflow/executors/multiple_executors.py
 ##
 @@ -0,0 +1,117 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Multiple  executor."""
+from itertools import chain
+from typing import Any, Dict, Iterable, Optional, Set
+
+from airflow.executors.base_executor import BaseExecutor, 
BaseExecutorProtocol, CommandType
+from airflow.models import TaskInstance
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstanceKeyType
+
+
+class MultipleExecutors(BaseExecutorProtocol):
+"""
+This executor can run multiple executors under the hood.
+"""
+def __init__(self, main_executor: BaseExecutor, additional_executors_dict: 
Dict[str, BaseExecutor]):
+super().__init__()
+self.main_executor = main_executor
+self.additional_executors_dict = additional_executors_dict
+self.executor_set: Set[BaseExecutor] = 
set(additional_executors_dict.values())
+self.executor_set.add(main_executor)
+self.log.info("Multiple executor")
+self.log.info("Main executor %s", str(main_executor))
+self.log.info("Additional executor dict %s", 
str(self.additional_executors_dict))
+self.log.info("Executor set %s", str(self.executor_set))
+
+def _get_executor_for_queue(self, queue: Optional[str]) -> BaseExecutor:
+executor = None
+if queue:
+executor = self.additional_executors_dict.get(queue)
+return self.main_executor if not executor else executor
+
+def start(self):
+for executor in self.executor_set:
+executor.start()
+
+def has_task(self, task_instance: TaskInstance) -> bool:
+executor = self._get_executor_for_queue(task_instance.queue)
+return executor.has_task(task_instance=task_instance)
+
+def sync(self) -> None:
+for executor in self.executor_set:
+executor.sync()
 
 Review comment:
   Yes. That's a great comment.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6702: [AIRFLOW-6140] Add missing types for some core classes.

mik-laj commented on a change in pull request #6702: [AIRFLOW-6140] Add missing 
types for some core classes.
URL: https://github.com/apache/airflow/pull/6702#discussion_r353706295
 
 

 ##
 File path: .travis.yml
 ##
 @@ -24,7 +24,7 @@ env:
 - MOUNT_LOCAL_SOURCES="false"
 - AIRFLOW_MOUNT_HOST_VOLUMES_FOR_STATIC_CHECKS="true"
 - FORCE_ANSWER_TO_QUESTIONS="yes"
-- AIRFLOW_CI_SILENT="true"
+- AIRFLOW_CI_SILENT="false"
 
 Review comment:
   It looks like unrelated changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #6715: [AIRFLOW-5945] Make inbuilt OperatorLinks work when using Serialization

kaxil commented on a change in pull request #6715: [AIRFLOW-5945] Make inbuilt 
OperatorLinks work when using Serialization
URL: https://github.com/apache/airflow/pull/6715#discussion_r353696421
 
 

 ##
 File path: tests/serialization/test_dag_serialization.py
 ##
 @@ -381,6 +362,49 @@ def test_roundtrip_relativedelta(self, val, expected):
 round_tripped = SerializedDAG._deserialize(serialized)
 self.assertEqual(val, round_tripped)
 
+def test_extra_serialized_field_and_operator_links(self):
+"""
+Assert extra field exists & OperatorLinks defined in Plugins and 
inbuilt Operator Links.
+
+This tests also depends on GoogleLink() registered as a plugin
+in tests/plugins/test_plugin.py
+
+The function tests that if extra operator links are registered in 
plugin
+in ``operator_extra_links`` and the same is also defined in
+the Operator in ``BaseOperator.operator_extra_links``, it has the 
correct
+extra link.
+"""
+test_date = datetime(2019, 8, 1)
+dag = DAG(dag_id='simple_dag', start_date=test_date)
+CustomBaseOperator(task_id='simple_task', dag=dag, bash_command="true")
+
+serialized_dag = SerializedDAG.to_dict(dag)
+self.assertIn("bash_command", serialized_dag["dag"]["tasks"][0])
+
+dag = SerializedDAG.from_dict(serialized_dag)
+simple_task = dag.task_dict["simple_task"]
+self.assertEqual(getattr(simple_task, "bash_command"), "true")
+
+#
+# Verify Operator Links work with Serialized Operator
+#
+# Check Serialized version of operator link only contains the inbuilt 
Op Link
+self.assertEqual(
+serialized_dag["dag"]["tasks"][0]["_operator_links_sources"],
+[{'airflow.utils.tests.CustomBaseOpLink': {}}]
 
 Review comment:
   Yes like:
   
   
https://github.com/apache/airflow/pull/6715/files#diff-5a167b261e34f0a91d371b5429ace188R492-R499
 :
   
   ```
   # Check Serialized version of operator link
   self.assertEqual(
   serialized_dag["dag"]["tasks"][0]["_operator_links_sources"],
   [
   
{'airflow.gcp.operators.bigquery.BigQueryConsoleIndexableLink': {'index': 0}},
   
{'airflow.gcp.operators.bigquery.BigQueryConsoleIndexableLink': {'index': 1}}
   ]
   )
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

ashb commented on a change in pull request #6655: [AIRFLOW-6056] Allow 
EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#discussion_r353693997
 
 

 ##
 File path: airflow/contrib/operators/emr_add_steps_operator.py
 ##
 @@ -28,33 +28,57 @@ class EmrAddStepsOperator(BaseOperator):
 
 :param job_flow_id: id of the JobFlow to add steps to. (templated)
 :type job_flow_id: str
+:param job_flow_name: name of the JobFlow to add steps to. Use as an 
alternative to passing
+job_flow_id. will search for id of JobFlow with matching name in one 
of the states in
+param cluster_states. Exactly one cluster like this should exist or 
will fail. (templated)
+:type job_flow_name: str
+:param cluster_states: Acceptable cluster states when searching for 
JobFlow id by param job_flow_name. (templated)
+:type cluster_states: list
 :param aws_conn_id: aws connection to uses
 :type aws_conn_id: str
 :param steps: boto3 style steps to be added to the jobflow. (templated)
 :type steps: list
+:param do_xcom_push: if True, job_flow_id is pushed to XCom with key 
job_flow_id.
+:type do_xcom_push: bool
 """
-template_fields = ['job_flow_id', 'steps']
+template_fields = ['job_flow_id', 'job_flow_name', 'cluster_states', 
'steps']
 template_ext = ()
 ui_color = '#f9c915'
 
 @apply_defaults
 def __init__(
-self,
-job_flow_id,
-aws_conn_id='aws_default',
-steps=None,
-*args, **kwargs):
+self,
+job_flow_id=None,
+job_flow_name=None,
+cluster_states=None,
+aws_conn_id='s3_default',
+steps=None,
+*args, **kwargs):
 
 Review comment:
   This formatting looks odd - please could you keep in indented as it was?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

ashb commented on a change in pull request #6655: [AIRFLOW-6056] Allow 
EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#discussion_r353696071
 
 

 ##
 File path: tests/contrib/operators/test_emr_add_steps_operator.py
 ##
 @@ -93,13 +101,30 @@ def test_render_template(self):
 def test_execute_returns_step_id(self):
 self.emr_client_mock.add_job_flow_steps.return_value = 
ADD_STEPS_SUCCESS_RETURN
 
-# Mock out the emr_client creator
-emr_session_mock = MagicMock()
-emr_session_mock.client.return_value = self.emr_client_mock
-self.boto3_session_mock = MagicMock(return_value=emr_session_mock)
+with patch('boto3.session.Session', self.boto3_session_mock):
+self.assertEqual(self.operator.execute(self.mock_context), 
['s-2LH3R5GW3A53T'])
+
+def test_init_with_cluster_name(self):
+expected_job_flow_id = 'j-1231231234'
+
+self.emr_client_mock.get_cluster_id_by_name.return_value = 
expected_job_flow_id
+self.emr_client_mock.add_job_flow_steps.return_value = 
ADD_STEPS_SUCCESS_RETURN
 
 with patch('boto3.session.Session', self.boto3_session_mock):
-self.assertEqual(self.operator.execute(None), ['s-2LH3R5GW3A53T'])
+operator = EmrAddStepsOperator(
+task_id='test_task',
+job_flow_name='test_cluster',
+cluster_states=['RUNNING', 'WAITING'],
+aws_conn_id='aws_default',
+dag=DAG('test_dag_id', default_args=self.args)
+)
+
+operator.execute(self.mock_context)
+
+ti = self.mock_context['ti']
+job_flow_id = ti.xcom_push.mock_calls[0][2]['value']
+
+self.assertEqual(job_flow_id, expected_job_flow_id)
 
 Review comment:
   This would be more clear as:
   
   ```python
   ti.xcom_push.assert_any_call(key='job_flow_id', 
value=expected_job_flow_id)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #6715: [AIRFLOW-5945] Make inbuilt OperatorLinks work when using Serialization

ashb commented on a change in pull request #6715: [AIRFLOW-5945] Make inbuilt 
OperatorLinks work when using Serialization
URL: https://github.com/apache/airflow/pull/6715#discussion_r353691980
 
 

 ##
 File path: tests/serialization/test_dag_serialization.py
 ##
 @@ -381,6 +362,49 @@ def test_roundtrip_relativedelta(self, val, expected):
 round_tripped = SerializedDAG._deserialize(serialized)
 self.assertEqual(val, round_tripped)
 
+def test_extra_serialized_field_and_operator_links(self):
+"""
+Assert extra field exists & OperatorLinks defined in Plugins and 
inbuilt Operator Links.
+
+This tests also depends on GoogleLink() registered as a plugin
+in tests/plugins/test_plugin.py
+
+The function tests that if extra operator links are registered in 
plugin
+in ``operator_extra_links`` and the same is also defined in
+the Operator in ``BaseOperator.operator_extra_links``, it has the 
correct
+extra link.
+"""
+test_date = datetime(2019, 8, 1)
+dag = DAG(dag_id='simple_dag', start_date=test_date)
+CustomBaseOperator(task_id='simple_task', dag=dag, bash_command="true")
+
+serialized_dag = SerializedDAG.to_dict(dag)
+self.assertIn("bash_command", serialized_dag["dag"]["tasks"][0])
+
+dag = SerializedDAG.from_dict(serialized_dag)
+simple_task = dag.task_dict["simple_task"]
+self.assertEqual(getattr(simple_task, "bash_command"), "true")
+
+#
+# Verify Operator Links work with Serialized Operator
+#
+# Check Serialized version of operator link only contains the inbuilt 
Op Link
+self.assertEqual(
+serialized_dag["dag"]["tasks"][0]["_operator_links_sources"],
+[{'airflow.utils.tests.CustomBaseOpLink': {}}]
 
 Review comment:
   What (if anything) is stored in the value (`{}`) -- is that are attributes 
of the operator link would go?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6725: [AIRFLOW-6157] Support for multiple executors

nuclearpinguin commented on a change in pull request #6725: [AIRFLOW-6157] 
Support for multiple executors
URL: https://github.com/apache/airflow/pull/6725#discussion_r353689795
 
 

 ##
 File path: airflow/executors/executor_loader.py
 ##
 @@ -31,47 +32,108 @@ class ExecutorLoader:
 DASK_EXECUTOR = "DaskExecutor"
 KUBERNETES_EXECUTOR = "KubernetesExecutor"
 
-_default_executor: Optional[BaseExecutor] = None
+_default_executor: Optional[BaseExecutorProtocol] = None
+
+_all_executors: Dict[str, BaseExecutor] = dict()
 
 @classmethod
-def get_default_executor(cls) -> BaseExecutor:
+def get_default_executor(cls) -> BaseExecutorProtocol:
 """Creates a new instance of the configured executor if none exists 
and returns it"""
 if cls._default_executor is not None:
 return cls._default_executor
 
 from airflow.configuration import conf
-executor_name = conf.get('core', 'EXECUTOR')
+main_executor_name = conf.get('core', 'EXECUTOR')
 
-cls._default_executor = ExecutorLoader._get_executor(executor_name)
+main_executor = cls._get_executor(main_executor_name)
 
+additional_executor_specs = conf.get('core', "ADDITIONAL_EXECUTORS")
+if additional_executor_specs:
+executor_array = additional_executor_specs.split(' ')
+additional_executors_dict: Dict[str, BaseExecutor] = {}
+for executor_spec in executor_array:
+executor_name, queue_array_string = executor_spec.split(':')
+executor = cls._get_executor(executor_name)
+queue_array = queue_array_string.split(",")
+for queue_name in queue_array:
+additional_executors_dict[queue_name] = executor
+cls._default_executor = MultipleExecutors(
+main_executor=main_executor,
+additional_executors_dict=additional_executors_dict)
+else:
+cls._default_executor = main_executor
 from airflow import LoggingMixin
 log = LoggingMixin().log
-log.info("Using executor %s", executor_name)
+if additional_executor_specs:
+log.info("Using MultipleExecutor with main executor %s and 
additional executors %s",
+ main_executor_name, additional_executor_specs)
+else:
+log.info("Using executor %s", main_executor_name)
 
 return cls._default_executor
 
+@classmethod
+def get_or_create_executor(cls, executor_name: str,
+   create_executor: Callable[[], BaseExecutor]) -> 
BaseExecutor:
+"""Retrieves (and creates if needed) an executor with the name 
specified"""
+if executor_name in cls._all_executors:
+return cls._all_executors[executor_name]
+executor = create_executor()
+cls._all_executors[executor_name] = executor
+return executor
+
+@staticmethod
+def create_local_executor() -> BaseExecutor:
+"""Creates LocalExecutor"""
+from airflow.executors.local_executor import LocalExecutor
+return LocalExecutor()
+
 @staticmethod
-def _get_executor(executor_name: str) -> BaseExecutor:
+def create_sequential_executor() -> BaseExecutor:
+"""Creates SequentialExecutor"""
+from airflow.executors.sequential_executor import SequentialExecutor
+return SequentialExecutor()
+
+@staticmethod
+def create_celery_executor() -> BaseExecutor:
+"""Creates CeleryExecutor"""
+from airflow.executors.celery_executor import CeleryExecutor
+return CeleryExecutor()
+
+@staticmethod
+def create_dask_executor() -> BaseExecutor:
+"""Creates DaskExecutor"""
+from airflow.executors.dask_executor import DaskExecutor
+return DaskExecutor()
+
+@staticmethod
+def create_kubernetes_executor() -> BaseExecutor:
+"""Creates KubernetesExecutor"""
+from airflow.executors.kubernetes_executor import KubernetesExecutor
+return KubernetesExecutor()
+
+@classmethod
+def _get_executor(cls, executor_name: str) -> BaseExecutor:
 """
 Creates a new instance of the named executor.
 In case the executor name is unknown in airflow,
-look for it in the plugins
+look for it in the plugins.
 """
 if executor_name == ExecutorLoader.LOCAL_EXECUTOR:
-from airflow.executors.local_executor import LocalExecutor
-return LocalExecutor()
+return cls.get_or_create_executor(executor_name=executor_name,
+  
create_executor=ExecutorLoader.create_local_executor)
 elif executor_name == ExecutorLoader.SEQUENTIAL_EXECUTOR:
-from airflow.executors.sequential_executor import 
SequentialExecutor
-return SequentialExecutor()
+return

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6725: [AIRFLOW-6157] Support for multiple executors

nuclearpinguin commented on a change in pull request #6725: [AIRFLOW-6157] 
Support for multiple executors
URL: https://github.com/apache/airflow/pull/6725#discussion_r353691352
 
 

 ##
 File path: airflow/executors/multiple_executors.py
 ##
 @@ -0,0 +1,117 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Multiple  executor."""
+from itertools import chain
+from typing import Any, Dict, Iterable, Optional, Set
+
+from airflow.executors.base_executor import BaseExecutor, 
BaseExecutorProtocol, CommandType
+from airflow.models import TaskInstance
+from airflow.models.taskinstance import SimpleTaskInstance, TaskInstanceKeyType
+
+
+class MultipleExecutors(BaseExecutorProtocol):
+"""
+This executor can run multiple executors under the hood.
+"""
+def __init__(self, main_executor: BaseExecutor, additional_executors_dict: 
Dict[str, BaseExecutor]):
+super().__init__()
+self.main_executor = main_executor
+self.additional_executors_dict = additional_executors_dict
+self.executor_set: Set[BaseExecutor] = 
set(additional_executors_dict.values())
+self.executor_set.add(main_executor)
+self.log.info("Multiple executor")
+self.log.info("Main executor %s", str(main_executor))
+self.log.info("Additional executor dict %s", 
str(self.additional_executors_dict))
+self.log.info("Executor set %s", str(self.executor_set))
+
+def _get_executor_for_queue(self, queue: Optional[str]) -> BaseExecutor:
+executor = None
+if queue:
+executor = self.additional_executors_dict.get(queue)
+return self.main_executor if not executor else executor
+
+def start(self):
+for executor in self.executor_set:
+executor.start()
+
+def has_task(self, task_instance: TaskInstance) -> bool:
+executor = self._get_executor_for_queue(task_instance.queue)
+return executor.has_task(task_instance=task_instance)
+
+def sync(self) -> None:
+for executor in self.executor_set:
+executor.sync()
 
 Review comment:
   I just wonder if we should use try / except in such loops? Because now when 
first executor fails everything fails. But using exception handling we can 
"execute as much as possible" and then throw an exception. WDYT?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #6715: [AIRFLOW-5945] Make inbuilt OperatorLinks work when using Serialization

ashb commented on a change in pull request #6715: [AIRFLOW-5945] Make inbuilt 
OperatorLinks work when using Serialization
URL: https://github.com/apache/airflow/pull/6715#discussion_r353690550
 
 

 ##
 File path: airflow/serialization/serialized_objects.py
 ##
 @@ -202,7 +203,11 @@ def _deserialize(cls, encoded_var):  # pylint: 
disable=too-many-return-statement
 return [cls._deserialize(v) for v in encoded_var]
 
 assert isinstance(encoded_var, dict)
-var = encoded_var[Encoding.VAR]
+try:
+var = encoded_var[Encoding.VAR]
+except KeyError as e:
+logging.info("encoded_var: %s", encoded_var)
 
 Review comment:
   What's this line for? (Being at info seems like the wrong level)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #6715: [AIRFLOW-5945] Make inbuilt OperatorLinks work when using Serialization

ashb commented on a change in pull request #6715: [AIRFLOW-5945] Make inbuilt 
OperatorLinks work when using Serialization
URL: https://github.com/apache/airflow/pull/6715#discussion_r353686706
 
 

 ##
 File path: airflow/contrib/operators/qubole_operator.py
 ##
 @@ -240,3 +241,9 @@ def __setattr__(self, name, value):
 self.kwargs[name] = value
 else:
 object.__setattr__(self, name, value)
+
+@classmethod
+def get_serialized_fields(cls):
+"""Serialized QuboleOperator contain exactly these fields."""
+cls._serialized_fields = frozenset(super().get_serialized_fields() | 
{"qubole_conn_id"})
 
 Review comment:
   Oh hang on - I've got an idea. So if you delcare a var called `__x` python 
automatically converts that to `_MyClass__x`, so we can do this:
   
   ```python
   class Base:
   def __init__(self, x=1):
   self.x = x
   
   __serialized_fields = None
   
   @classmethod
   def get_serialized_fields(cls):
   if not cls.__serialized_fields:
   cls.__serialized_fields = frozenset(vars(Base()))
   return cls.__serialized_fields
   
   
   class  Sub(Base):
   def __init__(self, y, **kwargs):
   super().__init__(**kwargs)
   self.y = y
   
   __serialized_fields = None
   @classmethod
   def get_serialized_fields(cls):
   if not cls.__serialized_fields:
   cls.__serialized_fields = super().get_serialized_fields() | 
frozenset({'y'})
   return cls.__serialized_fields
   
   
   print(Sub.get_serialized_fields())
   print(Base.get_serialized_fields())
   ```
   
   And then get as output
   
   ```
   frozenset({'x', 'y'})
   frozenset({'x'})
   ```
   
   
https://docs.python.org/3/tutorial/classes.html?highlight=class%20private#tut-private


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] baolsen commented on issue #6686: [AIRFLOW-6072] aws_hook: Outbound http proxy setting and other enhancements

baolsen commented on issue #6686: [AIRFLOW-6072] aws_hook: Outbound http proxy 
setting and other enhancements
URL: https://github.com/apache/airflow/pull/6686#issuecomment-561593830
 
 
   This build is failing due to some GCP error but none of the files I changes 
are GCP related.
   
   Main Airflow build is currently failing, I probably rebased from there so my 
build is failing. Will wait until the main Airflow build passes and then rebase 
this PR, which should fix this build.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #6715: [AIRFLOW-5945] Make inbuilt OperatorLinks work when using Serialization

2019-12-04 Thread ASF subversion and git services (Jira)

ashb commented on a change in pull request #6715: [AIRFLOW-5945] Make inbuilt 
OperatorLinks work when using Serialization
URL: https://github.com/apache/airflow/pull/6715#discussion_r353675344
 
 

 ##
 File path: airflow/serialization/serialized_baseoperator.py
 ##
 @@ -124,3 +142,50 @@ def _is_excluded(cls, var, attrname, op):
 # Don't store empty executor config or params dicts.
 return True
 return super()._is_excluded(var, attrname, op)
+
+@classmethod
+def _deserialize_operator_extra_links(
+cls,
+encoded_op_links: list
+) -> Dict[str, BaseOperatorLink]:
+"""
+Deserialize Operator Links by importing all the modules containing 
operator links
+defined in the serialized dict and initialize them by passing 
arguments using cattr.
+
+:param encoded_op_links: Serialized Operator Link
+:return: De-Serialized Operator Link
+"""
+
+# Extra Operator Links
+op_predefined_extra_links = {}
+
+for _operator_links_source in encoded_op_links:
+_operator_link_class, data = 
list(_operator_links_source.items())[0]
+try:
+single_link = import_string(_operator_link_class)
+op_predefined_extra_link: BaseOperatorLink = 
cattr.structure(data, single_link)
+op_predefined_extra_links.update(
+{op_predefined_extra_link.name: op_predefined_extra_link}
+)
+except ImportError:
+raise ImportError("Error Importing Operator Link: 
{}".format(_operator_link_class))
+
+return op_predefined_extra_links
+
+@classmethod
+def _serialize_operator_extra_links(
+cls,
+operator_extra_links: Iterable[BaseOperatorLink]
+):
+"""
+Serialize Operator Links. Store the import path of the OperatorLink 
and the arguments
+passed to it. Example 
``[{'airflow.gcp.operators.bigquery.BigQueryConsoleLink': {}}]``
+
+:param operator_extra_links: Operator Link
+:return: Serialized Operator Link
+"""
+return [
+{
+"{}.{}".format(x.__class__.__module__, x.__class__.__name__): 
cattr.unstructure(x)
 
 Review comment:
   What if someone wants to add a link via plugin that needs some attributes 
stored when (de)serializing?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-5902) local_task_job is sleeping more often than it needs and spamming task log

2019-12-04 Thread Ash Berlin-Taylor (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-5902.

Fix Version/s: 2.0.0
   Resolution: Fixed

> local_task_job is sleeping more often than it needs and spamming task log
> -
>
> Key: AIRFLOW-5902
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5902
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Minor
> Fix For: 2.0.0
>
>
> local_task_log's _execute loop is calling extra unnecessary sleep after a 
> heartbeat to keep up with the heart rate.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5902) local_task_job is sleeping more often than it needs and spamming task log



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987734#comment-16987734
 ] 

ASF subversion and git services commented on AIRFLOW-5902:
--

Commit dbf81df2205f36268745b76e575292c91be8 in airflow's branch 
refs/heads/master from Qingping Hou
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=dbf81df ]

[AIRFLOW-5902] avoid unnecessary sleep to maintain local task job heart rate 
(#6553)

sleep to maintain heart rate is already done by the hearbeat() call

> local_task_job is sleeping more often than it needs and spamming task log
> -
>
> Key: AIRFLOW-5902
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5902
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.0.0
>Reporter: QP Hou
>Assignee: QP Hou
>Priority: Minor
>
> local_task_log's _execute loop is calling extra unnecessary sleep after a 
> heartbeat to keep up with the heart rate.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ashb merged pull request #6553: [AIRFLOW-5902] avoid unnecessary sleep to maintain local task job heart rate

ashb merged pull request #6553: [AIRFLOW-5902] avoid unnecessary sleep to 
maintain local task job heart rate
URL: https://github.com/apache/airflow/pull/6553
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #6311: [AIRFLOW-5615] Reduce duplicated logic around job heartbeating

2019-12-04 Thread ASF subversion and git services (Jira)

ashb commented on issue #6311: [AIRFLOW-5615] Reduce duplicated logic around 
job heartbeating
URL: https://github.com/apache/airflow/pull/6311#issuecomment-561583718
 
 
   cc @houqp since you've been working on the code here too. (Yes, I need to 
fix up these tests)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6095) Task stats endpoint is slow and returns a large payload



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987726#comment-16987726
 ] 

ASF subversion and git services commented on AIRFLOW-6095:
--

Commit 527c0dd94408e69e5070e11b528fd457196449e6 in airflow's branch 
refs/heads/master from Robin Edwards
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=527c0dd ]

[AIRFLOW-6095] Filter dags returned by task_stats (#6684)

Add dag_ids parameter to task_stats so can filter by a set of dag_ids
present on the dags view. This is intended to speed up the response time
and reduce the size of the payload when running a large number of dags.

> Task stats endpoint is slow and returns a large payload
> ---
>
> Key: AIRFLOW-6095
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6095
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: api, ui
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Robin Edwards
>Assignee: Robin Edwards
>Priority: Minor
> Fix For: 1.10.7
>
>
> The task_stats endpoint returns all dags by default. This can result in an 
> extremely large payload ~ 8mb and slow response time when you have a lot of 
> dags (In this case 1500+).
> The accompanying pull request adds a dag_ids get parameter to the task_stats 
> end point which is populated by the dags present on the page.
> If this is accepted I intend to provide the equivalent change to the 
> dag_stats and last_dagruns endpoints



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6095) Task stats endpoint is slow and returns a large payload

2019-12-04 Thread Ash Berlin-Taylor (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-6095.

Fix Version/s: 1.10.7
   Resolution: Fixed

> Task stats endpoint is slow and returns a large payload
> ---
>
> Key: AIRFLOW-6095
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6095
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: api, ui
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Robin Edwards
>Assignee: Robin Edwards
>Priority: Minor
> Fix For: 1.10.7
>
>
> The task_stats endpoint returns all dags by default. This can result in an 
> extremely large payload ~ 8mb and slow response time when you have a lot of 
> dags (In this case 1500+).
> The accompanying pull request adds a dag_ids get parameter to the task_stats 
> end point which is populated by the dags present on the page.
> If this is accepted I intend to provide the equivalent change to the 
> dag_stats and last_dagruns endpoints



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6095) Task stats endpoint is slow and returns a large payload



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987725#comment-16987725
 ] 

ASF GitHub Bot commented on AIRFLOW-6095:
-

ashb commented on pull request #6684: [AIRFLOW-6095] Filter dags returned by 
task_stats
URL: https://github.com/apache/airflow/pull/6684
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Task stats endpoint is slow and returns a large payload
> ---
>
> Key: AIRFLOW-6095
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6095
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: api, ui
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Robin Edwards
>Assignee: Robin Edwards
>Priority: Minor
>
> The task_stats endpoint returns all dags by default. This can result in an 
> extremely large payload ~ 8mb and slow response time when you have a lot of 
> dags (In this case 1500+).
> The accompanying pull request adds a dag_ids get parameter to the task_stats 
> end point which is populated by the dags present on the page.
> If this is accepted I intend to provide the equivalent change to the 
> dag_stats and last_dagruns endpoints



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ashb commented on issue #6684: [AIRFLOW-6095] Filter dags returned by task_stats

ashb commented on issue #6684: [AIRFLOW-6095] Filter dags returned by task_stats
URL: https://github.com/apache/airflow/pull/6684#issuecomment-561582484
 
 
   /cc @KevinYang21 You might find this useful.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb merged pull request #6684: [AIRFLOW-6095] Filter dags returned by task_stats

ashb merged pull request #6684: [AIRFLOW-6095] Filter dags returned by 
task_stats
URL: https://github.com/apache/airflow/pull/6684
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] aviemzur commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id

aviemzur commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to 
accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655#issuecomment-561573195
 
 
   @ashb Added commit with the changes you requested.
   
   Changed the operator to push `job_flow_id` to `xcom` so it can be used to 
pass to other operators for example `EmrStepSensor`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Comment Edited] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration

2019-12-04 Thread Bjorn Olsen (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987709#comment-16987709
 ] 

Bjorn Olsen edited comment on AIRFLOW-6083 at 12/4/19 10:15 AM:


This should be resolved with AIRFLOW-6072

After this PR gets merged you could use the "Extra" part of the Airflow 
Connection to pass additional config to the boto3 client / resource / session 
as you please.

 

Edit: You should also be able to change the timeout this way, if desired.


was (Author: bjorn.ols...@gmail.com):
This should be resolved with AIRFLOW-6072

After this PR gets merged you could use the "Extra" part of the Airflow 
Connection to pass additional config to the boto3 client / resource / session 
as you please.

> AwsLambdaHook is not accepting non-default configuration
> 
>
> Key: AIRFLOW-6083
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6083
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: aws
>Affects Versions: 1.10.6
>Reporter: Daumantas Pagojus
>Assignee: Daumantas Pagojus
>Priority: Blocker
>
> Hello.
> While using Airflow we have come across a problem with AwsLambdaHook.
> We are using this hook to launch Lambda function which takes a while to 
> complete (around 5-6 minutes). However, since lambda is invoked through boto3 
> client, which has default timeout set to [60 
> seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/],
>  our Airflow interface shows this lambda failing, even though Lambda finishes 
> successfully checking at AWS console. This also causes another side effect: 
> Since boto3 thinks that Lambda has timed out, it automatically spawns another 
> instance, which also times out and this chain lasts 5 times, spawning 5 
> Lambdas and all these Lambdas show as failed in Airflow interface, while they 
> actually succeed.
>  
> This can be solved by passing in custom configuration when creating a boto3 
> client, however, it is not possible to do that when creating AwsLambdaHook as 
> it does not take in this parameter.
> However, we see that AwsLambdaHook inherits and uses AwsHook's function 
> (get_client_type) to get the boto3 client and this function accepts 
> configuration parameter (which defaults to None), but it is never passed to 
> it from the Lambda's hook, which could be easily achieved and would fix the 
> bug we are facing at the moment.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6083) AwsLambdaHook is not accepting non-default configuration

2019-12-04 Thread Bjorn Olsen (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987709#comment-16987709
 ] 

Bjorn Olsen commented on AIRFLOW-6083:
--

This should be resolved with AIRFLOW-6072

After this PR gets merged you could use the "Extra" part of the Airflow 
Connection to pass additional config to the boto3 client / resource / session 
as you please.

> AwsLambdaHook is not accepting non-default configuration
> 
>
> Key: AIRFLOW-6083
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6083
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: aws
>Affects Versions: 1.10.6
>Reporter: Daumantas Pagojus
>Assignee: Daumantas Pagojus
>Priority: Blocker
>
> Hello.
> While using Airflow we have come across a problem with AwsLambdaHook.
> We are using this hook to launch Lambda function which takes a while to 
> complete (around 5-6 minutes). However, since lambda is invoked through boto3 
> client, which has default timeout set to [60 
> seconds|https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/],
>  our Airflow interface shows this lambda failing, even though Lambda finishes 
> successfully checking at AWS console. This also causes another side effect: 
> Since boto3 thinks that Lambda has timed out, it automatically spawns another 
> instance, which also times out and this chain lasts 5 times, spawning 5 
> Lambdas and all these Lambdas show as failed in Airflow interface, while they 
> actually succeed.
>  
> This can be solved by passing in custom configuration when creating a boto3 
> client, however, it is not possible to do that when creating AwsLambdaHook as 
> it does not take in this parameter.
> However, we see that AwsLambdaHook inherits and uses AwsHook's function 
> (get_client_type) to get the boto3 client and this function accepts 
> configuration parameter (which defaults to None), but it is never passed to 
> it from the Lambda's hook, which could be easily achieved and would fix the 
> bug we are facing at the moment.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AIRFLOW-4355) Externally triggered DAG is marked as 'success' even if a task has been 'removed'!

2019-12-04 Thread Bjorn Olsen (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bjorn Olsen reassigned AIRFLOW-4355:


Assignee: Bjorn Olsen

> Externally triggered DAG is marked as 'success' even if a task has been 
> 'removed'!
> --
>
> Key: AIRFLOW-4355
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4355
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, DagRun, scheduler
>Affects Versions: 1.10.3
>Reporter: t oo
>Assignee: Bjorn Olsen
>Priority: Blocker
>  Labels: dynamic
> Fix For: 2.0.0
>
> Attachments: dag_success_even_if_task_removed.png, treeview.png
>
>
> note: all my dags are purely externally triggered
> *Issue:* Dag has 5 parallel tasks that ran successfully and 1 final task that 
> somehow got 'removed' state (prior dag runs had 'failed' state) and never ran 
> successfully but still the DAG is showing success!
>  
> *Command ran* (note that previous commands like airflow trigger_dag -e 
> 20190412 qsr_coremytbl were run before and failed for valid reason (ie python 
> task failing) ):
> airflow trigger_dag -e 20190412T08:00 qsr_coremytbl --conf '\{"hourstr":"08"}'
>  
> *some logs on prior instance of airflow (ec2 was autohealed):*
> [2019-04-18 08:29:40,678] \{logging_mixin.py:95} INFO - [2019-04-18 
> 08:29:40,678] {__init__.py:4897} WARNING - Failed to get task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00 [None]>' 
> for dag ''. Marking it as removed.
>  [2019-04-18 08:29:43,582] \{logging_mixin.py:95} INFO - [2019-04-18 
> 08:29:43,582] {__init__.py:4906} INFO - Restoring task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00 [removed]>' 
> which was previously removed from DAG ''
>  [2019-04-18 08:29:43,618] \{jobs.py:1787} INFO - Creating / updating 
>  08:00:00+00:00 [scheduled]> in ORM
>  [2019-04-18 08:29:43,676] \{logging_mixin.py:95} INFO - [2019-04-18 
> 08:29:43,676] {__init__.py:4897} WARNING - Failed to get task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00 
> [scheduled]>' for dag ''. Marking it as removed.
>  
> *some logs on newer ec2:*
> [myuser@host logs]$ grep -i hive -R * | sed 's#[0-9]#x#g' | sort | uniq -c | 
> grep -v 'airflow-webserver-access.log'
>  2 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx  qsr_coremytbl 
> REPAIR_HIVE_schemeh.mytbl log -xx-xx xx:xx:xx.xx rsawyerx 
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id', 
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
>  1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx  qsr_coremytbl 
> REPAIR_HIVE_schemeh.mytbl log -xx-xx xx:xx:xx.xx rsawyerx 
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id', 
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl'), ('format', 
> u'json')]
>  1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx  qsr_coremytbl 
> REPAIR_HIVE_schemeh.mytbl rendered -xx-xx xx:xx:xx.xx rsawyerx 
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id', 
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
>  1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx  qsr_coremytbl 
> REPAIR_HIVE_schemeh.mytbl task -xx-xx xx:xx:xx.xx rsawyerx 
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id', 
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
>  1 scheduler/latest/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx] 
> \{jobs.py:} INFO - Creating / updating  qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl -xx-xx xx:xx:xx+xx:xx 
> [scheduled]> in ORM
>  71 scheduler/latest/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx] 
> \{logging_mixin.py:xx} INFO - [-xx-xx xx:xx:xx,xxx] {__init__.py:} 
> INFO - Restoring task ' -xx-xx xx:xx:xx+xx:xx [removed]>' which was previously removed from DAG 
> ''
>  1 scheduler/-xx-xx/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx] 
> \{jobs.py:} INFO - Creating / updating  qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl -xx-xx xx:xx:xx+xx:xx 
> [scheduled]> in ORM
>  71 scheduler/-xx-xx/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx] 
> \{logging_mixin.py:xx} INFO - [-xx-xx xx:xx:xx,xxx] {__init__.py:} 
> INFO - Restoring task ' -xx-xx xx:xx:xx+xx:xx [removed]>' which was previously removed from DAG 
> ''
>  
> mysql> *select * from task_instance where task_id like '%REP%';#*
>  
>

[GitHub] [airflow] potiuk commented on issue #6585: More GSOD improvements