date:20190912

mik-laj commented on issue #5765: [AIRFLOW-4412] PodLauncher drops log lines 
printed by the pod
URL: https://github.com/apache/airflow/pull/5765#issuecomment-531118328
 
 
   can you say why the project was postponed?  is it due to implementation 
problems? What problems have arisen?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #5765: [AIRFLOW-4412] PodLauncher drops log lines printed by the pod

mik-laj commented on issue #5765: [AIRFLOW-4412] PodLauncher drops log lines 
printed by the pod
URL: https://github.com/apache/airflow/pull/5765#issuecomment-531117877
 
 
   No problem. I will do it. Thanks you.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] smrt28 commented on issue #5765: [AIRFLOW-4412] PodLauncher drops log lines printed by the pod

smrt28 commented on issue #5765: [AIRFLOW-4412] PodLauncher drops log lines 
printed by the pod
URL: https://github.com/apache/airflow/pull/5765#issuecomment-531095222
 
 
   @mik-laj I spent a hour fixing those broken tests. I realized it would be 
too big effort for me to investigate how all those tests work and how to fix 
them. Please, feel free to take the ticket and finish it. The project which 
uses airflow in my company was postponed and I have no time and even the 
environment for fixing this. Sorry.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] bryan831 opened a new pull request #6089: add Dentsu to README

bryan831 opened a new pull request #6089: add Dentsu to README
URL: https://github.com/apache/airflow/pull/6089
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] clacke edited a comment on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill

clacke edited a comment on issue #5498: [AIRFLOW-4509] SubDagOperator using 
scheduler instead of backfill
URL: https://github.com/apache/airflow/pull/5498#issuecomment-531052581
 
 
   The head of the sub-DAG itself has no explicit dependencies, the only 
constraint is that it should only be run when the SubDagOperator asks it to.
   
   I will try with latest master today.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] clacke commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill

clacke commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler 
instead of backfill
URL: https://github.com/apache/airflow/pull/5498#issuecomment-531052581
 
 
   The head of the sub-DAG itself has no dependencies, the only constraint is 
that it should only be run when the SubDagOperator asks it to.
   
   I will try with latest master today.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work started] (AIRFLOW-5469) Airflow CLI render error on unicode in the stdout



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-5469 started by Cong Zhao.
--
> Airflow CLI render error on unicode in the stdout
> -
>
> Key: AIRFLOW-5469
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5469
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli
>Affects Versions: 1.10.0
>Reporter: Cong Zhao
>Assignee: Cong Zhao
>Priority: Major
> Fix For: 1.10.6
>
>
> As a non-English user, lots of my hive scripts come along with unicode 
> characters as string values. The hive queries render correctly on Web, but 
> not in CLI. Below is the log of the render output:
> {quote}/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
> DeprecationWarning: Importing 'HivePartitionSensor' directly from 
> 'airflow.operators' has been deprecated. Please import from 
> 'airflow.operators.[operator_module]' instead. Support for direct imports 
> will be dropped entirely in Airflow 2.0.
> DeprecationWarning)
> /usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
> DeprecationWarning: Importing 'EmailOperator' directly from 
> 'airflow.operators' has been deprecated. Please import from 
> 'airflow.operators.[operator_module]' instead. Support for direct imports 
> will be dropped entirely in Airflow 2.0.
> DeprecationWarning)
> /usr/local/lib/python2.7/dist-packages/airflow/models.py:4007: 
> PendingDeprecationWarning: The requested task could not be added to the DAG 
> because a task with task_id emails_done is already in the DAG. Starting in 
> Airflow 2.0, trying to overwrite a task will raise an exception.
> category=PendingDeprecationWarning)
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in 
> args.func(args)
> File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74, 
> in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 601, 
> in render
> """.format(attr, getattr(task, attr
> UnicodeEncodeError: 'ascii' codec can't encode characters in position 
> 2138-2139: ordinal not in range(128)
> exitCode 1
> {quote}
>  The issue is simply the string format in the render function doesn't support 
> unicode. 
> [https://github.com/apache/airflow/blob/1.10.0/airflow/bin/cli.py#L603]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5469) Airflow CLI render error on unicode in the stdout



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cong Zhao updated AIRFLOW-5469:
---
Description: 
As a non-English user, lots of my hive scripts come along with unicode 
characters as string values. The hive queries render correctly on Web, but not 
in CLI. Below is the log of the render output:
{quote}/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
DeprecationWarning: Importing 'HivePartitionSensor' directly from 
'airflow.operators' has been deprecated. Please import from 
'airflow.operators.[operator_module]' instead. Support for direct imports will 
be dropped entirely in Airflow 2.0.

DeprecationWarning)

/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
DeprecationWarning: Importing 'EmailOperator' directly from 'airflow.operators' 
has been deprecated. Please import from 'airflow.operators.[operator_module]' 
instead. Support for direct imports will be dropped entirely in Airflow 2.0.

DeprecationWarning)

/usr/local/lib/python2.7/dist-packages/airflow/models.py:4007: 
PendingDeprecationWarning: The requested task could not be added to the DAG 
because a task with task_id emails_done is already in the DAG. Starting in 
Airflow 2.0, trying to overwrite a task will raise an exception.

category=PendingDeprecationWarning)

Traceback (most recent call last):

File "/usr/local/bin/airflow", line 32, in 

args.func(args)

File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74, in 
wrapper

return f(*args, **kwargs)

File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 601, in 
render

""".format(attr, getattr(task, attr

UnicodeEncodeError: 'ascii' codec can't encode characters in position 
2138-2139: ordinal not in range(128)

exitCode 1
{quote}
 The issue is simply the string format in the render function doesn't support 
unicode. [https://github.com/apache/airflow/blob/1.10.0/airflow/bin/cli.py#L603]

  was:
As a non-English user, lots of my hive scripts come along with unicode 
characters as string values. The hive queries render correctly on Web, but not 
in CLI. Below is the log of the render output:

{quote}
 /usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
DeprecationWarning: Importing 'HivePartitionSensor' directly from 
'airflow.operators' has been deprecated. Please import from 
'airflow.operators.[operator_module]' instead. Support for direct imports will 
be dropped entirely in Airflow 2.0.

DeprecationWarning)

/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
DeprecationWarning: Importing 'EmailOperator' directly from 'airflow.operators' 
has been deprecated. Please import from 'airflow.operators.[operator_module]' 
instead. Support for direct imports will be dropped entirely in Airflow 2.0.

DeprecationWarning)

/usr/local/lib/python2.7/dist-packages/airflow/models.py:4007: 
PendingDeprecationWarning: The requested task could not be added to the DAG 
because a task with task_id emails_done is already in the DAG. Starting in 
Airflow 2.0, trying to overwrite a task will raise an exception.

category=PendingDeprecationWarning)

Traceback (most recent call last):

File "/usr/local/bin/airflow", line 32, in 

args.func(args)

File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74, in 
wrapper

return f(*args, **kwargs)

File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 601, in 
render

""".format(attr, getattr(task, attr

UnicodeEncodeError: 'ascii' codec can't encode characters in position 
2138-2139: ordinal not in range(128)

exitCode 1
{quote}


> Airflow CLI render error on unicode in the stdout
> -
>
> Key: AIRFLOW-5469
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5469
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli
>Affects Versions: 1.10.0
>Reporter: Cong Zhao
>Assignee: Cong Zhao
>Priority: Major
> Fix For: 1.10.6
>
>
> As a non-English user, lots of my hive scripts come along with unicode 
> characters as string values. The hive queries render correctly on Web, but 
> not in CLI. Below is the log of the render output:
> {quote}/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
> DeprecationWarning: Importing 'HivePartitionSensor' directly from 
> 'airflow.operators' has been deprecated. Please import from 
> 'airflow.operators.[operator_module]' instead. Support for direct imports 
> will be dropped entirely in Airflow 2.0.
> DeprecationWarning)
> /usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
> DeprecationWarning: Importing 'EmailOperator' directly from 
> 'airflow.operators' has been deprecated. Please import from 
> 'airflow.operators.[operator_module]' instead. Support for dir

[jira] [Updated] (AIRFLOW-5469) Airflow CLI render error on unicode in the stdout



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cong Zhao updated AIRFLOW-5469:
---
Description: 
As a non-English user, lots of my hive scripts come along with unicode 
characters as string values. The hive queries render correctly on Web, but not 
in CLI. Below is the log of the render output:

{quote}
 /usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
DeprecationWarning: Importing 'HivePartitionSensor' directly from 
'airflow.operators' has been deprecated. Please import from 
'airflow.operators.[operator_module]' instead. Support for direct imports will 
be dropped entirely in Airflow 2.0.

DeprecationWarning)

/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
DeprecationWarning: Importing 'EmailOperator' directly from 'airflow.operators' 
has been deprecated. Please import from 'airflow.operators.[operator_module]' 
instead. Support for direct imports will be dropped entirely in Airflow 2.0.

DeprecationWarning)

/usr/local/lib/python2.7/dist-packages/airflow/models.py:4007: 
PendingDeprecationWarning: The requested task could not be added to the DAG 
because a task with task_id emails_done is already in the DAG. Starting in 
Airflow 2.0, trying to overwrite a task will raise an exception.

category=PendingDeprecationWarning)

Traceback (most recent call last):

File "/usr/local/bin/airflow", line 32, in 

args.func(args)

File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74, in 
wrapper

return f(*args, **kwargs)

File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 601, in 
render

""".format(attr, getattr(task, attr

UnicodeEncodeError: 'ascii' codec can't encode characters in position 
2138-2139: ordinal not in range(128)

exitCode 1
{quote}

  was:
As a non-English user, lots of my hive scripts come along with unicode 
characters as string values. The hive queries render correctly on Web, but not 
in CLI. Below is the log of the render output:
```


> Airflow CLI render error on unicode in the stdout
> -
>
> Key: AIRFLOW-5469
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5469
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli
>Affects Versions: 1.10.0
>Reporter: Cong Zhao
>Assignee: Cong Zhao
>Priority: Major
> Fix For: 1.10.6
>
>
> As a non-English user, lots of my hive scripts come along with unicode 
> characters as string values. The hive queries render correctly on Web, but 
> not in CLI. Below is the log of the render output:
> {quote}
>  /usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
> DeprecationWarning: Importing 'HivePartitionSensor' directly from 
> 'airflow.operators' has been deprecated. Please import from 
> 'airflow.operators.[operator_module]' instead. Support for direct imports 
> will be dropped entirely in Airflow 2.0.
> DeprecationWarning)
> /usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:385: 
> DeprecationWarning: Importing 'EmailOperator' directly from 
> 'airflow.operators' has been deprecated. Please import from 
> 'airflow.operators.[operator_module]' instead. Support for direct imports 
> will be dropped entirely in Airflow 2.0.
> DeprecationWarning)
> /usr/local/lib/python2.7/dist-packages/airflow/models.py:4007: 
> PendingDeprecationWarning: The requested task could not be added to the DAG 
> because a task with task_id emails_done is already in the DAG. Starting in 
> Airflow 2.0, trying to overwrite a task will raise an exception.
> category=PendingDeprecationWarning)
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in 
> args.func(args)
> File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74, 
> in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 601, 
> in render
> """.format(attr, getattr(task, attr
> UnicodeEncodeError: 'ascii' codec can't encode characters in position 
> 2138-2139: ordinal not in range(128)
> exitCode 1
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (AIRFLOW-5469) Airflow CLI render error on unicode in the stdout

Cong Zhao created AIRFLOW-5469:
--

 Summary: Airflow CLI render error on unicode in the stdout
 Key: AIRFLOW-5469
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5469
 Project: Apache Airflow
  Issue Type: Bug
  Components: cli
Affects Versions: 1.10.0
Reporter: Cong Zhao
Assignee: Cong Zhao
 Fix For: 1.10.6


As a non-English user, lots of my hive scripts come along with unicode 
characters as string values. The hive queries render correctly on Web, but not 
in CLI. Below is the log of the render output:
```



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AIRFLOW-5349) Add schedulername option for KubernetesPodOperator



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928922#comment-16928922
 ] 

ASF GitHub Bot commented on AIRFLOW-5349:
-

antonymayi commented on pull request #6088: [AIRFLOW-5349] Add schedulername 
option for KubernetesPodOperator
URL: https://github.com/apache/airflow/pull/6088
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-5349
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Kubernetes PODs might be [specified with explicit 
scheduler](https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/#specify-schedulers-for-pods).
 This feature, however, is currently not available from the 
`KubernetesPodOperator` API.
   
   I am offering simple PR that adds this option via a `schedulername` 
parameter.
   
   ### Tests
   
   - [x] My PR adds the following unit tests:
  - `tests.minikube.test_kubernetes_pod_operator.test_pod_schedulername`
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add schedulername option for KubernetesPodOperator
> --
>
> Key: AIRFLOW-5349
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5349
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: api, configuration, contrib
>Affects Versions: 1.10.4
>Reporter: Antony Mayi
>Priority: Major
>  Labels: pull-request-available
>
> Kubernetes PODs might be [specified with explicit 
> scheduler|https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/#specify-schedulers-for-pods].
>  This feature, however, is currently not available from the 
> {{KubernetesPodOperator}} API.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (AIRFLOW-5349) Add schedulername option for KubernetesPodOperator

2019-09-12 Thread Antony Mayi (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918960#comment-16918960
 ] 

Antony Mayi edited comment on AIRFLOW-5349 at 9/12/19 11:02 PM:


Implemented per [PR #6088|https://github.com/apache/airflow/pull/6088].


was (Author: antonymayi):
Implemented per [PR #6015|https://github.com/apache/airflow/pull/6015].

> Add schedulername option for KubernetesPodOperator
> --
>
> Key: AIRFLOW-5349
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5349
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: api, configuration, contrib
>Affects Versions: 1.10.4
>Reporter: Antony Mayi
>Priority: Major
>  Labels: pull-request-available
>
> Kubernetes PODs might be [specified with explicit 
> scheduler|https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/#specify-schedulers-for-pods].
>  This feature, however, is currently not available from the 
> {{KubernetesPodOperator}} API.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AIRFLOW-5349) Add schedulername option for KubernetesPodOperator



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928924#comment-16928924
 ] 

ASF GitHub Bot commented on AIRFLOW-5349:
-

antonymayi commented on pull request #6015: [AIRFLOW-5349] Added schedulername 
option for KubernetesPodOperator
URL: https://github.com/apache/airflow/pull/6015
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add schedulername option for KubernetesPodOperator
> --
>
> Key: AIRFLOW-5349
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5349
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: api, configuration, contrib
>Affects Versions: 1.10.4
>Reporter: Antony Mayi
>Priority: Major
>  Labels: pull-request-available
>
> Kubernetes PODs might be [specified with explicit 
> scheduler|https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/#specify-schedulers-for-pods].
>  This feature, however, is currently not available from the 
> {{KubernetesPodOperator}} API.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [airflow] antonymayi commented on issue #6015: [AIRFLOW-5349] Added schedulername option for KubernetesPodOperator

antonymayi commented on issue #6015: [AIRFLOW-5349] Added schedulername option 
for KubernetesPodOperator
URL: https://github.com/apache/airflow/pull/6015#issuecomment-531038562
 
 
   I am replacing this PR with #6088


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] antonymayi closed pull request #6015: [AIRFLOW-5349] Added schedulername option for KubernetesPodOperator

antonymayi closed pull request #6015: [AIRFLOW-5349] Added schedulername option 
for KubernetesPodOperator
URL: https://github.com/apache/airflow/pull/6015
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] antonymayi opened a new pull request #6088: [AIRFLOW-5349] Add schedulername option for KubernetesPodOperator

antonymayi opened a new pull request #6088: [AIRFLOW-5349] Add schedulername
option for KubernetesPodOperator
URL: https://github.com/apache/airflow/pull/6088

Make sure you have checked _all_ steps below.

### Jira

- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5349

### Description

- [x] Here are some details about my PR, including screenshots of any UI
changes:

Kubernetes PODs might be [specified with explicit
scheduler](https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/#specify-schedulers-for-pods).
This feature, however, is currently not available from the
`KubernetesPodOperator` API.

I am offering simple PR that adds this option via a `schedulername`
parameter.

### Tests

- [x] My PR adds the following unit tests:
- `tests.minikube.test_kubernetes_pod_operator.test_pod_schedulername`

### Commits

- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Documentation

- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release

### Code Quality

- [x] Passes `flake8`

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #5281: WIP: [AIRFLOW-4506] Make GIT_SYNC_DEPTH and GIT_SYNC_REV configurable

mik-laj commented on issue #5281: WIP: [AIRFLOW-4506] Make GIT_SYNC_DEPTH and 
GIT_SYNC_REV configurable
URL: https://github.com/apache/airflow/pull/5281#issuecomment-531037032
 
 
   @andrejbaran  Can I help you with this change?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #5680: [AIRFLOW-5066] allow k8s fieldref substitution

mik-laj commented on issue #5680: [AIRFLOW-5066] allow k8s fieldref substitution
URL: https://github.com/apache/airflow/pull/5680#issuecomment-531036060
 
 
   @jsurloppe  Can you do rebase? I would like to merge these changes 
   
   CC: @potiuk 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj edited a comment on issue #5680: [AIRFLOW-5066] allow k8s fieldref substitution

mik-laj edited a comment on issue #5680: [AIRFLOW-5066] allow k8s fieldref 
substitution
URL: https://github.com/apache/airflow/pull/5680#issuecomment-531036060
 
 
   @jsurloppe  Can you do rebase? I would like to merge these changes 
   
   @potiuk ping


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #5694: [AIRFLOW-5082] add subject in aws sns hook

mik-laj commented on a change in pull request #5694: [AIRFLOW-5082] add subject 
in aws sns hook
URL: https://github.com/apache/airflow/pull/5694#discussion_r323980061
 
 

 ##
 File path: tests/contrib/hooks/test_aws_sns_hook.py
 ##
 @@ -42,9 +42,10 @@ def test_publish_to_target(self):
 
 message = "Hello world"
 topic_name = "test-topic"
+subject = "test-subject"
 target = hook.get_conn().create_topic(Name=topic_name).get('TopicArn')
 
-response = hook.publish_to_target(target, message)
+response = hook.publish_to_target(target, message, subject)
 
 Review comment:
   we have an additional condition in 
https://github.com/apache/airflow/pull/5694/files#diff-287babf794e581d216b0122119867824R58.
 Can you add a separate test?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #5694: [AIRFLOW-5082] add subject in aws sns hook

mik-laj commented on a change in pull request #5694: [AIRFLOW-5082] add subject 
in aws sns hook
URL: https://github.com/apache/airflow/pull/5694#discussion_r323979807
 
 

 ##
 File path: airflow/contrib/hooks/aws_sns_hook.py
 ##
 @@ -53,8 +55,16 @@ def publish_to_target(self, target_arn, message):
 'default': message
 }
 
+if subject is None:
 
 Review comment:
   Does this method work without this "if"?  For me it should work when the 
subject is equal to None. Very often, the default method arguments are None.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #5765: [AIRFLOW-4412] PodLauncher drops log lines printed by the pod

mik-laj commented on issue #5765: [AIRFLOW-4412] PodLauncher drops log lines 
printed by the pod
URL: https://github.com/apache/airflow/pull/5765#issuecomment-531034561
 
 
   Hi. Could you do rebase? I also planned to deal with this problem. 
https://github.com/PolideaInternal/airflow/issues/268


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #6015: [AIRFLOW-5349] Added schedulername option for KubernetesPodOperator

2019-09-12 Thread ASF subversion and git services (Jira)

mik-laj commented on issue #6015: [AIRFLOW-5349] Added schedulername option for 
KubernetesPodOperator
URL: https://github.com/apache/airflow/pull/6015#issuecomment-531033084
 
 
   Hi. This PR appears to contain unrelated changes. Could you look at it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5449) Can't commit blank values on variable edits



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928915#comment-16928915
 ] 

ASF subversion and git services commented on AIRFLOW-5449:
--

Commit 29860b7e7f4516ca72e37facf47595b0107decb1 in airflow's branch 
refs/heads/v1-10-stable from Andrew Desousa
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=29860b7 ]

[AIRFLOW-5449] Users can now set existing variable values to None (#6084)



> Can't commit blank values on variable edits
> ---
>
> Key: AIRFLOW-5449
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5449
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration
>Affects Versions: 1.10.5
>Reporter: Andrew Desousa
>Assignee: Andrew Desousa
>Priority: Minor
>
> On Airflow, when going to "admin >> variables" and attempting to change the 
> value of an existing variable to nothing, the changes aren't being set 
> properly and saving the edit does nothing. This bug fix should allow users to 
> blank out the values of existing variables.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [airflow] mik-laj merged pull request #6084: [AIRFLOW-5449] Users can now set existing variable values to None

mik-laj merged pull request #6084: [AIRFLOW-5449] Users can now set existing 
variable values to None
URL: https://github.com/apache/airflow/pull/6084
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5449) Can't commit blank values on variable edits



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928914#comment-16928914
 ] 

ASF GitHub Bot commented on AIRFLOW-5449:
-

mik-laj commented on pull request #6084: [AIRFLOW-5449] Users can now set 
existing variable values to None
URL: https://github.com/apache/airflow/pull/6084
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Can't commit blank values on variable edits
> ---
>
> Key: AIRFLOW-5449
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5449
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration
>Affects Versions: 1.10.5
>Reporter: Andrew Desousa
>Assignee: Andrew Desousa
>Priority: Minor
>
> On Airflow, when going to "admin >> variables" and attempting to change the 
> value of an existing variable to nothing, the changes aren't being set 
> properly and saving the edit does nothing. This bug fix should allow users to 
> blank out the values of existing variables.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AIRFLOW-5447) KubernetesExecutor hangs on task queueing



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928907#comment-16928907
 ] 

Henry Cohen commented on AIRFLOW-5447:
--

[~dimberman]p py3

> KubernetesExecutor hangs on task queueing
> -
>
> Key: AIRFLOW-5447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5447
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: executor-kubernetes
>Affects Versions: 1.10.4, 1.10.5
> Environment: Kubernetes version v1.14.3, Airflow version 1.10.4-1.10.5
>Reporter: Henry Cohen
>Assignee: Daniel Imberman
>Priority: Blocker
>
> Starting in 1.10.4, and continuing in 1.10.5, when using the 
> KubernetesExecutor, with the webserver and scheduler running in the 
> kubernetes cluster, tasks are scheduled, but when added to the task queue, 
> the executor process hangs indefinitely. Based on log messages, it appears to 
> be stuck at this line 
> https://github.com/apache/airflow/blob/v1-10-stable/airflow/contrib/executors/kubernetes_executor.py#L761



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [airflow] mik-laj commented on a change in pull request #5998: [AIRFLOW-5398] Update contrib example DAGs to context manager

mik-laj commented on a change in pull request #5998: [AIRFLOW-5398] Update 
contrib example DAGs to context manager
URL: https://github.com/apache/airflow/pull/5998#discussion_r323966106
 
 

 ##
 File path: airflow/contrib/example_dags/example_dingding_operator.py
 ##
 @@ -59,173 +60,163 @@ def failure_callback(context):
 args['on_failure_callback'] = failure_callback
 # [END howto_operator_dingding_failure_callback]
 
-dag = DAG(
+with DAG(
 dag_id='example_dingding_operator',
 default_args=args,
 schedule_interval='@once',
-dagrun_timeout=timedelta(minutes=60),
-)
-
-# [START howto_operator_dingding]
-text_msg_remind_none = DingdingOperator(
-task_id='text_msg_remind_none',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind none',
-at_mobiles=None,
-at_all=False,
-dag=dag,
-)
-# [END howto_operator_dingding]
-
-text_msg_remind_specific = DingdingOperator(
-task_id='text_msg_remind_specific',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind specific users',
-at_mobiles=['156', '130'],
-at_all=False,
-dag=dag,
-)
-
-text_msg_remind_include_invalid = DingdingOperator(
-task_id='text_msg_remind_include_invalid',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind users including invalid',
-# 123 is invalid user or user not in the group
-at_mobiles=['156', '123'],
-at_all=False,
-dag=dag,
-)
-
-# [START howto_operator_dingding_remind_users]
-text_msg_remind_all = DingdingOperator(
-task_id='text_msg_remind_all',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind all users in group',
-# list of user phone/email here in the group
-# when at_all is specific will cover at_mobiles
-at_mobiles=['156', '130'],
-at_all=True,
-dag=dag,
-)
-# [END howto_operator_dingding_remind_users]
-
-link_msg = DingdingOperator(
-task_id='link_msg',
-dingding_conn_id='dingding_default',
-message_type='link',
-message={
-'title': 'Airflow dingding link message',
-'text': 'Airflow official documentation link',
-'messageUrl': 'http://airflow.apache.org',
-'picURL': 'http://airflow.apache.org/_images/pin_large.png'
-},
-dag=dag,
-)
-
-# [START howto_operator_dingding_rich_text]
-markdown_msg = DingdingOperator(
-task_id='markdown_msg',
-dingding_conn_id='dingding_default',
-message_type='markdown',
-message={
-'title': 'Airflow dingding markdown message',
-'text': '# Markdown message title\n'
-'content content .. \n'
-'### sub-title\n'
-'![logo](http://airflow.apache.org/_images/pin_large.png)'
-},
-at_mobiles=['156'],
-at_all=False,
-dag=dag,
-)
-# [END howto_operator_dingding_rich_text]
-
-single_action_card_msg = DingdingOperator(
-task_id='single_action_card_msg',
-dingding_conn_id='dingding_default',
-message_type='actionCard',
-message={
-'title': 'Airflow dingding single actionCard message',
-'text': 'Airflow dingding single actionCard message\n'
-'![logo](http://airflow.apache.org/_images/pin_large.png)\n'
-'This is a official logo in Airflow website.',
-'hideAvatar': '0',
-'btnOrientation': '0',
-'singleTitle': 'read more',
-'singleURL': 'http://airflow.apache.org'
-},
-dag=dag,
-)
-
-multi_action_card_msg = DingdingOperator(
-task_id='multi_action_card_msg',
-dingding_conn_id='dingding_default',
-message_type='actionCard',
-message={
-'title': 'Airflow dingding multi actionCard message',
-'text': 'Airflow dingding multi actionCard message\n'
-'![logo](http://airflow.apache.org/_images/pin_large.png)\n'
-'Airflow documentation and github',
-'hideAvatar': '0',
-'btnOrientation': '0',
-'btns': [
-{
-'title': 'Airflow Documentation',
-'actionURL': 'http://airflow.apache.org'
-},
-{
-'title': 'Airflow Github',
-'actionURL': 'https://github.com/apache/airflow'
-}
-]
-},
-dag=dag,
-)
-
-feed_card_msg = DingdingOperator(
-task_id='feed_card_msg',
-dingding_conn_id='dingding_default',
-message_type='feedCard',
-message={
-"links": [
-{
-"title": "Airflow DAG feed card",
-"messageURL": 
"https://airflow.readthedocs.io/en/latest/ui.html";,
-"picURL": "http://airflow.apache.org/_images/dags.png";
-},
-{
-"title": "Airflow tree feed

[GitHub] [airflow] mik-laj commented on a change in pull request #5998: [AIRFLOW-5398] Update contrib example DAGs to context manager

mik-laj commented on a change in pull request #5998: [AIRFLOW-5398] Update 
contrib example DAGs to context manager
URL: https://github.com/apache/airflow/pull/5998#discussion_r323965847
 
 

 ##
 File path: airflow/contrib/example_dags/example_dingding_operator.py
 ##
 @@ -59,173 +60,163 @@ def failure_callback(context):
 args['on_failure_callback'] = failure_callback
 # [END howto_operator_dingding_failure_callback]
 
-dag = DAG(
+with DAG(
 dag_id='example_dingding_operator',
 default_args=args,
 schedule_interval='@once',
-dagrun_timeout=timedelta(minutes=60),
-)
-
-# [START howto_operator_dingding]
-text_msg_remind_none = DingdingOperator(
-task_id='text_msg_remind_none',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind none',
-at_mobiles=None,
-at_all=False,
-dag=dag,
-)
-# [END howto_operator_dingding]
-
-text_msg_remind_specific = DingdingOperator(
-task_id='text_msg_remind_specific',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind specific users',
-at_mobiles=['156', '130'],
-at_all=False,
-dag=dag,
-)
-
-text_msg_remind_include_invalid = DingdingOperator(
-task_id='text_msg_remind_include_invalid',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind users including invalid',
-# 123 is invalid user or user not in the group
-at_mobiles=['156', '123'],
-at_all=False,
-dag=dag,
-)
-
-# [START howto_operator_dingding_remind_users]
-text_msg_remind_all = DingdingOperator(
-task_id='text_msg_remind_all',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind all users in group',
-# list of user phone/email here in the group
-# when at_all is specific will cover at_mobiles
-at_mobiles=['156', '130'],
-at_all=True,
-dag=dag,
-)
-# [END howto_operator_dingding_remind_users]
-
-link_msg = DingdingOperator(
-task_id='link_msg',
-dingding_conn_id='dingding_default',
-message_type='link',
-message={
-'title': 'Airflow dingding link message',
-'text': 'Airflow official documentation link',
-'messageUrl': 'http://airflow.apache.org',
-'picURL': 'http://airflow.apache.org/_images/pin_large.png'
-},
-dag=dag,
-)
-
-# [START howto_operator_dingding_rich_text]
-markdown_msg = DingdingOperator(
-task_id='markdown_msg',
-dingding_conn_id='dingding_default',
-message_type='markdown',
-message={
-'title': 'Airflow dingding markdown message',
-'text': '# Markdown message title\n'
-'content content .. \n'
-'### sub-title\n'
-'![logo](http://airflow.apache.org/_images/pin_large.png)'
-},
-at_mobiles=['156'],
-at_all=False,
-dag=dag,
-)
-# [END howto_operator_dingding_rich_text]
-
-single_action_card_msg = DingdingOperator(
-task_id='single_action_card_msg',
-dingding_conn_id='dingding_default',
-message_type='actionCard',
-message={
-'title': 'Airflow dingding single actionCard message',
-'text': 'Airflow dingding single actionCard message\n'
-'![logo](http://airflow.apache.org/_images/pin_large.png)\n'
-'This is a official logo in Airflow website.',
-'hideAvatar': '0',
-'btnOrientation': '0',
-'singleTitle': 'read more',
-'singleURL': 'http://airflow.apache.org'
-},
-dag=dag,
-)
-
-multi_action_card_msg = DingdingOperator(
-task_id='multi_action_card_msg',
-dingding_conn_id='dingding_default',
-message_type='actionCard',
-message={
-'title': 'Airflow dingding multi actionCard message',
-'text': 'Airflow dingding multi actionCard message\n'
-'![logo](http://airflow.apache.org/_images/pin_large.png)\n'
-'Airflow documentation and github',
-'hideAvatar': '0',
-'btnOrientation': '0',
-'btns': [
-{
-'title': 'Airflow Documentation',
-'actionURL': 'http://airflow.apache.org'
-},
-{
-'title': 'Airflow Github',
-'actionURL': 'https://github.com/apache/airflow'
-}
-]
-},
-dag=dag,
-)
-
-feed_card_msg = DingdingOperator(
-task_id='feed_card_msg',
-dingding_conn_id='dingding_default',
-message_type='feedCard',
-message={
-"links": [
-{
-"title": "Airflow DAG feed card",
-"messageURL": 
"https://airflow.readthedocs.io/en/latest/ui.html";,
-"picURL": "http://airflow.apache.org/_images/dags.png";
-},
-{
-"title": "Airflow tree feed

[GitHub] [airflow] mik-laj commented on a change in pull request #5998: [AIRFLOW-5398] Update contrib example DAGs to context manager

mik-laj commented on a change in pull request #5998: [AIRFLOW-5398] Update 
contrib example DAGs to context manager
URL: https://github.com/apache/airflow/pull/5998#discussion_r323965803
 
 

 ##
 File path: airflow/contrib/example_dags/example_dingding_operator.py
 ##
 @@ -59,173 +60,163 @@ def failure_callback(context):
 args['on_failure_callback'] = failure_callback
 # [END howto_operator_dingding_failure_callback]
 
-dag = DAG(
+with DAG(
 dag_id='example_dingding_operator',
 default_args=args,
 schedule_interval='@once',
-dagrun_timeout=timedelta(minutes=60),
-)
-
-# [START howto_operator_dingding]
-text_msg_remind_none = DingdingOperator(
-task_id='text_msg_remind_none',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind none',
-at_mobiles=None,
-at_all=False,
-dag=dag,
-)
-# [END howto_operator_dingding]
-
-text_msg_remind_specific = DingdingOperator(
-task_id='text_msg_remind_specific',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind specific users',
-at_mobiles=['156', '130'],
-at_all=False,
-dag=dag,
-)
-
-text_msg_remind_include_invalid = DingdingOperator(
-task_id='text_msg_remind_include_invalid',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind users including invalid',
-# 123 is invalid user or user not in the group
-at_mobiles=['156', '123'],
-at_all=False,
-dag=dag,
-)
-
-# [START howto_operator_dingding_remind_users]
-text_msg_remind_all = DingdingOperator(
-task_id='text_msg_remind_all',
-dingding_conn_id='dingding_default',
-message_type='text',
-message='Airflow dingding text message remind all users in group',
-# list of user phone/email here in the group
-# when at_all is specific will cover at_mobiles
-at_mobiles=['156', '130'],
-at_all=True,
-dag=dag,
-)
-# [END howto_operator_dingding_remind_users]
-
-link_msg = DingdingOperator(
-task_id='link_msg',
-dingding_conn_id='dingding_default',
-message_type='link',
-message={
-'title': 'Airflow dingding link message',
-'text': 'Airflow official documentation link',
-'messageUrl': 'http://airflow.apache.org',
-'picURL': 'http://airflow.apache.org/_images/pin_large.png'
-},
-dag=dag,
-)
-
-# [START howto_operator_dingding_rich_text]
-markdown_msg = DingdingOperator(
-task_id='markdown_msg',
-dingding_conn_id='dingding_default',
-message_type='markdown',
-message={
-'title': 'Airflow dingding markdown message',
-'text': '# Markdown message title\n'
-'content content .. \n'
-'### sub-title\n'
-'![logo](http://airflow.apache.org/_images/pin_large.png)'
-},
-at_mobiles=['156'],
-at_all=False,
-dag=dag,
-)
-# [END howto_operator_dingding_rich_text]
-
-single_action_card_msg = DingdingOperator(
-task_id='single_action_card_msg',
-dingding_conn_id='dingding_default',
-message_type='actionCard',
-message={
-'title': 'Airflow dingding single actionCard message',
-'text': 'Airflow dingding single actionCard message\n'
-'![logo](http://airflow.apache.org/_images/pin_large.png)\n'
-'This is a official logo in Airflow website.',
-'hideAvatar': '0',
-'btnOrientation': '0',
-'singleTitle': 'read more',
-'singleURL': 'http://airflow.apache.org'
-},
-dag=dag,
-)
-
-multi_action_card_msg = DingdingOperator(
-task_id='multi_action_card_msg',
-dingding_conn_id='dingding_default',
-message_type='actionCard',
-message={
-'title': 'Airflow dingding multi actionCard message',
-'text': 'Airflow dingding multi actionCard message\n'
-'![logo](http://airflow.apache.org/_images/pin_large.png)\n'
-'Airflow documentation and github',
-'hideAvatar': '0',
-'btnOrientation': '0',
-'btns': [
-{
-'title': 'Airflow Documentation',
-'actionURL': 'http://airflow.apache.org'
-},
-{
-'title': 'Airflow Github',
-'actionURL': 'https://github.com/apache/airflow'
-}
-]
-},
-dag=dag,
-)
-
-feed_card_msg = DingdingOperator(
-task_id='feed_card_msg',
-dingding_conn_id='dingding_default',
-message_type='feedCard',
-message={
-"links": [
-{
-"title": "Airflow DAG feed card",
-"messageURL": 
"https://airflow.readthedocs.io/en/latest/ui.html";,
-"picURL": "http://airflow.apache.org/_images/dags.png";
-},
-{
-"title": "Airflow tree feed

[jira] [Commented] (AIRFLOW-5447) KubernetesExecutor hangs on task queueing



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928901#comment-16928901
 ] 

Daniel Imberman commented on AIRFLOW-5447:
--

[~HPCohen] [~Yuval.Itzchakov] also question: Are you running py2 or py3? 

> KubernetesExecutor hangs on task queueing
> -
>
> Key: AIRFLOW-5447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5447
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: executor-kubernetes
>Affects Versions: 1.10.4, 1.10.5
> Environment: Kubernetes version v1.14.3, Airflow version 1.10.4-1.10.5
>Reporter: Henry Cohen
>Assignee: Daniel Imberman
>Priority: Blocker
>
> Starting in 1.10.4, and continuing in 1.10.5, when using the 
> KubernetesExecutor, with the webserver and scheduler running in the 
> kubernetes cluster, tasks are scheduled, but when added to the task queue, 
> the executor process hangs indefinitely. Based on log messages, it appears to 
> be stuck at this line 
> https://github.com/apache/airflow/blob/v1-10-stable/airflow/contrib/executors/kubernetes_executor.py#L761



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [airflow] mik-laj opened a new pull request #6087: [AIRFLOW-XXX] Display GCP integration in table

mik-laj opened a new pull request #6087: [AIRFLOW-XXX] Display GCP integration 
in table
URL: https://github.com/apache/airflow/pull/6087
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj edited a comment on issue #6086: [AIRFLOW-XXX] Display GCP integration in table

mik-laj edited a comment on issue #6086: [AIRFLOW-XXX] Display GCP integration 
in table
URL: https://github.com/apache/airflow/pull/6086#issuecomment-530956896
 
 
   @kaxil This file contains a guide that describes the [BigQuery Data Transfer 
Service](https://cloud.google.com/bigquery/transfer/). Unfortunately, we don't 
have a guide for the [BigQuery](https://cloud.google.com/bigquery/) yet.  I am 
trying to synchronize file names with guides, hooks and operators.
   https://user-images.githubusercontent.com/12058428/64812170-f72d4900-d59e-11e9-947c-7209343a99ea.png";>
   
   
   ```
  * - `BigQuery `__
-
- :mod:`airflow.gcp.hooks.bigquery`
- :mod:`airflow.gcp.operators.bigquery`
- :mod:`airflow.gcp.sensors.bigquery`
   
  * - `BigQuery Data Transfer Service 
`__
- :doc:`How to use `
- :mod:`airflow.gcp.hooks.bigquery_dts`
- :mod:`airflow.gcp.operators.bigquery_dts`
- :mod:`airflow.gcp.sensors.bigquery_dts`
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj edited a comment on issue #6086: [AIRFLOW-XXX] Display GCP integration in table

mik-laj edited a comment on issue #6086: [AIRFLOW-XXX] Display GCP integration 
in table
URL: https://github.com/apache/airflow/pull/6086#issuecomment-530956896
 
 
   @kaxil This file contains a guide that describes the [BigQuery Data Transfer 
Service](https://cloud.google.com/bigquery/transfer/). Unfortunately, we don't 
have a guide for [BigQuery](https://cloud.google.com/bigquery/) yet.  I am 
trying to synchronize file names with guides, hooks and operators.
   https://user-images.githubusercontent.com/12058428/64812170-f72d4900-d59e-11e9-947c-7209343a99ea.png";>
   
   
   ```
  * - `BigQuery `__
-
- :mod:`airflow.gcp.hooks.bigquery`
- :mod:`airflow.gcp.operators.bigquery`
- :mod:`airflow.gcp.sensors.bigquery`
   
  * - `BigQuery Data Transfer Service 
`__
- :doc:`How to use `
- :mod:`airflow.gcp.hooks.bigquery_dts`
- :mod:`airflow.gcp.operators.bigquery_dts`
- :mod:`airflow.gcp.sensors.bigquery_dts`
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] pingzh removed a comment on issue #6023: [AIRFLOW-5417] Fix DB disconnects during webserver startup

pingzh removed a comment on issue #6023: [AIRFLOW-5417] Fix DB disconnects 
during webserver startup
URL: https://github.com/apache/airflow/pull/6023#issuecomment-530958711
 
 
   cherry picked in `rb1.10.4` 
https://git.musta.ch/airbnb/airflow-internal/pull/307


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] pingzh commented on issue #6023: [AIRFLOW-5417] Fix DB disconnects during webserver startup

pingzh commented on issue #6023: [AIRFLOW-5417] Fix DB disconnects during 
webserver startup
URL: https://github.com/apache/airflow/pull/6023#issuecomment-530958711
 
 
   cherry picked in `rb1.10.4` 
https://git.musta.ch/airbnb/airflow-internal/pull/307


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj edited a comment on issue #6086: [AIRFLOW-XXX] Display GCP integration in table

mik-laj edited a comment on issue #6086: [AIRFLOW-XXX] Display GCP integration 
in table
URL: https://github.com/apache/airflow/pull/6086#issuecomment-530956896
 
 
   @kaxil This file contains a guide that describes the [BigQuery Data Transfer 
Service](https://cloud.google.com/bigquery/transfer/). Unfortunately, we don't 
have a guide for [Bigquery](https://cloud.google.com/bigquery/) yet.  I am 
trying to synchronize file names with guides, hooks and operators.
   https://user-images.githubusercontent.com/12058428/64812170-f72d4900-d59e-11e9-947c-7209343a99ea.png";>
   
   
   ```
  * - `BigQuery `__
-
- :mod:`airflow.gcp.hooks.bigquery`
- :mod:`airflow.gcp.operators.bigquery`
- :mod:`airflow.gcp.sensors.bigquery`
   
  * - `BigQuery Data Transfer Service 
`__
- :doc:`How to use `
- :mod:`airflow.gcp.hooks.bigquery_dts`
- :mod:`airflow.gcp.operators.bigquery_dts`
- :mod:`airflow.gcp.sensors.bigquery_dts`
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #6086: [AIRFLOW-XXX] Display GCP integration in table

mik-laj commented on issue #6086: [AIRFLOW-XXX] Display GCP integration in table
URL: https://github.com/apache/airflow/pull/6086#issuecomment-530956896
 
 
   @kaxil Because this file contains a guide that describes the [BigQuery Data 
Transfer Service](https://cloud.google.com/bigquery/transfer/). Unfortunately, 
we don't have a guide for [Bigquery](https://cloud.google.com/bigquery/).  I am 
trying to synchronize file names with guides, hooks and operators.
   
   ```
  * - `BigQuery `__
-
- :mod:`airflow.gcp.hooks.bigquery`
- :mod:`airflow.gcp.operators.bigquery`
- :mod:`airflow.gcp.sensors.bigquery`
   
  * - `BigQuery Data Transfer Service 
`__
- :doc:`How to use `
- :mod:`airflow.gcp.hooks.bigquery_dts`
- :mod:`airflow.gcp.operators.bigquery_dts`
- :mod:`airflow.gcp.sensors.bigquery_dts`
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj edited a comment on issue #6086: [AIRFLOW-XXX] Display GCP integration in table

mik-laj edited a comment on issue #6086: [AIRFLOW-XXX] Display GCP integration 
in table
URL: https://github.com/apache/airflow/pull/6086#issuecomment-530956896
 
 
   @kaxil This file contains a guide that describes the [BigQuery Data Transfer 
Service](https://cloud.google.com/bigquery/transfer/). Unfortunately, we don't 
have a guide for [Bigquery](https://cloud.google.com/bigquery/).  I am trying 
to synchronize file names with guides, hooks and operators.
   
   ```
  * - `BigQuery `__
-
- :mod:`airflow.gcp.hooks.bigquery`
- :mod:`airflow.gcp.operators.bigquery`
- :mod:`airflow.gcp.sensors.bigquery`
   
  * - `BigQuery Data Transfer Service 
`__
- :doc:`How to use `
- :mod:`airflow.gcp.hooks.bigquery_dts`
- :mod:`airflow.gcp.operators.bigquery_dts`
- :mod:`airflow.gcp.sensors.bigquery_dts`
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj merged pull request #6047: [AIRFLOW-XXX] Simplify AWS/Azure/Databricks operators listing

mik-laj merged pull request #6047: [AIRFLOW-XXX] Simplify AWS/Azure/Databricks 
operators listing
URL: https://github.com/apache/airflow/pull/6047
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6086: [AIRFLOW-XXX] Display GCP integration in table

codecov-io commented on issue #6086: [AIRFLOW-XXX] Display GCP integration in 
table
URL: https://github.com/apache/airflow/pull/6086#issuecomment-530950111
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6086?src=pr&el=h1) 
Report
   > Merging 
[#6086](https://codecov.io/gh/apache/airflow/pull/6086?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/19e2f2c7943646d3f7be31f85aafbe6854876d0c?src=pr&el=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6086/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6086?src=pr&el=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#6086   +/-   ##
   ===
 Coverage   80.09%   80.09%   
   ===
 Files 606  606   
 Lines   3489334893   
   ===
 Hits2794927949   
 Misses   6944 6944
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6086?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6086?src=pr&el=footer). 
Last update 
[19e2f2c...ee4fa72](https://codecov.io/gh/apache/airflow/pull/6086?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5447) KubernetesExecutor hangs on task queueing



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928804#comment-16928804
 ] 

Henry Cohen commented on AIRFLOW-5447:
--

This line in particular is what lead me through my investigation to 
https://github.com/apache/airflow/blob/v1-10-stable/airflow/contrib/executors/kubernetes_executor.py#L761:
{noformat}
[2019-09-12 17:56:05,186] kubernetes_executor.py:764 INFO - Add task 
('example_subdag_operator', 'start', datetime.datetime(2019, 9, 10, 0, 0, 
tzinfo=), 1) with command ['airflow', 
'run', 'example_subdag_operator', 'start', '2019-09-10T00:00:00+00:00', 
'--local', '--pool', 'default_pool', '-sd', 
'/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_subdag_operator.py']
 with executor_config {}{noformat}

> KubernetesExecutor hangs on task queueing
> -
>
> Key: AIRFLOW-5447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5447
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: executor-kubernetes
>Affects Versions: 1.10.4, 1.10.5
> Environment: Kubernetes version v1.14.3, Airflow version 1.10.4-1.10.5
>Reporter: Henry Cohen
>Assignee: Daniel Imberman
>Priority: Blocker
>
> Starting in 1.10.4, and continuing in 1.10.5, when using the 
> KubernetesExecutor, with the webserver and scheduler running in the 
> kubernetes cluster, tasks are scheduled, but when added to the task queue, 
> the executor process hangs indefinitely. Based on log messages, it appears to 
> be stuck at this line 
> https://github.com/apache/airflow/blob/v1-10-stable/airflow/contrib/executors/kubernetes_executor.py#L761



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (AIRFLOW-5447) KubernetesExecutor hangs on task queueing



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928788#comment-16928788
 ] 

Henry Cohen edited comment on AIRFLOW-5447 at 9/12/19 6:00 PM:
---

This is a sample of what I see when running with the example DAGs, they queue, 
but when the first one tries to start it just sits, and eventually the 
processes die and the scheduler hangs
{noformat}
[2019-09-12 17:56:03,034] kubernetes_executor.py:698 INFO - TaskInstance: 
 found in 
queued state but was not launched, rescheduling
 [2019-09-12 17:56:03,043] scheduler_job.py:1376 INFO - Resetting orphaned 
tasks for active dag runs
 [2019-09-12 17:56:03,085] base_job.py:308 INFO - Reset the following 30 
TaskInstances:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 [2019-09-12 17:56:03,092] dag_processing.py:545 INFO - Launched 
DagFileProcessorManager with pid: 35
 [2019-09-12 17:56:03,093] scheduler_job.py:1390 DEBUG - Starting Loop...
 [2019-09-12 17:56:03,093] scheduler_job.py:1401 DEBUG - Harvesting DAG parsing 
results
 [2019-09-12 17:56:03,093] scheduler_job.py:1403 DEBUG - Harvested 0 SimpleDAGs
 [2019-09-12 17:56:03,093] scheduler_job.py:1438 DEBUG - Heartbeating the 
executor
 [2019-09-12 17:56:03,093] base_executor.py:124 DEBUG - 0 running task instances
 [2019-09-12 17:56:03,094] base_executor.py:125 DEBUG - 0 in queue
 [2019-09-12 17:56:03,094] base_executor.py:126 DEBUG - 96 open slots
 [2019-09-12 17:56:03,094] base_executor.py:135 DEBUG - Calling the  sync method
 [2019-09-12 17:56:03,100] scheduler_job.py:1459 DEBUG - Ran scheduling loop in 
0.01 seconds
 [2019-09-12 17:56:03,101] scheduler_job.py:1462 DEBUG - Sleeping for 1.00 
seconds
 [2019-09-12 17:56:03,107] settings.py:54 INFO - Configured default timezone 

 [2019-09-12 17:56:03,109] settings.py:327 DEBUG - Failed to import 
airflow_local_settings.
 Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/airflow/settings.py", line 315, 
in import_local_settings
 import airflow_local_settings
 ModuleNotFoundError: No module named 'airflow_local_settings'
 [2019-09-12 17:56:03,111] logging_config.py:47 INFO - Successfully imported 
user-defined logging config from log_config.LOGGING_CONFIG
 [2019-09-12 17:56:03,120] settings.py:170 DEBUG - Setting up DB connection 
pool (PID 35)
 [2019-09-12 17:56:03,121] settings.py:213 INFO - settings.configure_orm(): 
Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=35
 [2019-09-12 17:56:03,289] settings.py:238 DEBUG - Disposing DB connection pool 
(PID 45)
 [2019-09-12 17:56:03,356] settings.py:238 DEBUG - Disposing DB connection pool 
(PID 41)
 [2019-09-12 17:56:04,101] scheduler_job.py:1474 DEBUG - Sleeping for 0.99 
seconds to prevent excessive logging
 [2019-09-12 17:56:04,126] scheduler_job.py:257 DEBUG - Waiting for 

 [2019-09-12 17:56:04,127] scheduler_job.py:257 DEBUG - Waiting for 

 [2019-09-12 17:56:04,162] settings.py:238 DEBUG - Disposing DB connection pool 
(PID 55)
 [2019-09-12 17:56:04,223] settings.py:238 DEBUG - Disposing DB connection pool 
(PID 58)
 [2019-09-12 17:56:05,095] scheduler_job.py:1390 DEBUG - Starting Loop...
 [2019-09-12 17:56:05,095] scheduler_job.py:1401 DEBUG - Harvesting DAG parsing 
results
 [2019-09-12 17:56:05,097] dag_processing.py:637 DEBUG - Received message of 
type DagParsingStat
 [2019-09-12 17:56:05,098] dag_processing.py:637 DEBUG - Received message of 
type SimpleDag
 [2019-09-12 17:56:05,098] dag_processing.py:637 DEBUG - Received message of 
type SimpleDag
 [2019-09-12 17:56:05,099] dag_processing.py:637 DEBUG - Received message of 
type SimpleDag
 [2019-09-12 17:56:05,099] dag_processing.py:637 DEBUG - Received message of 
type SimpleDag
 [2019-09-12 17:56:05,100] dag_processing.py:637 DEBUG - Received message of 
type DagParsingStat
 [2019-09-12 17:56:05,101] dag_processing.py:637 DEBUG - Received message of 
type DagParsingStat
 [2019-09-12 17:56:05,101] scheduler_job.py:1403 DEBUG - Harvested 4 SimpleDAGs
 [2019-09-12 17:56:05,128] scheduler_job.py:921 INFO - 5 tasks up for execution:
 
 
 
 
 
 [2019-09-12 17:56:05,138] scheduler_job.py:953 INFO - Figuring out tasks to 
run in Pool(name=default_pool) with 128 open slots and 5 task instances ready 
to be queued
 [2019-09-12 17:56:05,139] scheduler_job.py:981 INFO - DAG 
example_subdag_operator has 0/48 running and queued tasks
 [2019-09-12 17:56:05,139] scheduler_job.py:981 INFO - DAG 
latest_only_with_trigger has 0/48 running and queued tasks
 [2019-09-12 17:56:05,139] scheduler_job.py:981 INFO - DAG 
latest_only_with_trigger has 1/48 running and queued tasks
 [2019-09-12 17:56:05,139] scheduler_job.py:981 INFO - DAG 
latest_only_with_trigger has 2/48 running and queued tasks
 [2019-09-12 17:56:05,139] scheduler_job.py:981 INFO - DAG 
latest_only_with_trigger has 3/48 running and queued tasks
 [2019-09-12 17:56:05,139] schedule

[jira] [Comment Edited] (AIRFLOW-5447) KubernetesExecutor hangs on task queueing



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928788#comment-16928788
 ] 

Henry Cohen edited comment on AIRFLOW-5447 at 9/12/19 5:59 PM:
---

`[2019-09-12 17:56:03,034] {{kubernetes_executor.py:698}} INFO - TaskInstance: 
 found in 
queued state but was not launched, rescheduling
 [2019-09-12 17:56:03,043] {{scheduler_job.py:1376}} INFO - Resetting orphaned 
tasks for active dag runs
 [2019-09-12 17:56:03,085] {{base_job.py:308}} INFO - Reset the following 30 
TaskInstances:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 [2019-09-12 17:56:03,092] {{dag_processing.py:545}} INFO - Launched 
DagFileProcessorManager with pid: 35
 [2019-09-12 17:56:03,093] {{scheduler_job.py:1390}} DEBUG - Starting Loop...
 [2019-09-12 17:56:03,093] {{scheduler_job.py:1401}} DEBUG - Harvesting DAG 
parsing results
 [2019-09-12 17:56:03,093] {{scheduler_job.py:1403}} DEBUG - Harvested 0 
SimpleDAGs
 [2019-09-12 17:56:03,093] {{scheduler_job.py:1438}} DEBUG - Heartbeating the 
executor
 [2019-09-12 17:56:03,093] {{base_executor.py:124}} DEBUG - 0 running task 
instances
 [2019-09-12 17:56:03,094] {{base_executor.py:125}} DEBUG - 0 in queue
 [2019-09-12 17:56:03,094] {{base_executor.py:126}} DEBUG - 96 open slots
 [2019-09-12 17:56:03,094] {{base_executor.py:135}} DEBUG - Calling the  sync method
 [2019-09-12 17:56:03,100] {{scheduler_job.py:1459}} DEBUG - Ran scheduling 
loop in 0.01 seconds
 [2019-09-12 17:56:03,101] {{scheduler_job.py:1462}} DEBUG - Sleeping for 1.00 
seconds
 [2019-09-12 17:56:03,107] {{settings.py:54}} INFO - Configured default 
timezone 
 [2019-09-12 17:56:03,109] {{settings.py:327}} DEBUG - Failed to import 
airflow_local_settings.
 Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/airflow/settings.py", line 315, 
in import_local_settings
 import airflow_local_settings
 ModuleNotFoundError: No module named 'airflow_local_settings'
 [2019-09-12 17:56:03,111] {{logging_config.py:47}} INFO - Successfully 
imported user-defined logging config from log_config.LOGGING_CONFIG
 [2019-09-12 17:56:03,120] {{settings.py:170}} DEBUG - Setting up DB connection 
pool (PID 35)
 [2019-09-12 17:56:03,121] {{settings.py:213}} INFO - settings.configure_orm(): 
Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=35
 [2019-09-12 17:56:03,289] {{settings.py:238}} DEBUG - Disposing DB connection 
pool (PID 45)
 [2019-09-12 17:56:03,356] {{settings.py:238}} DEBUG - Disposing DB connection 
pool (PID 41)
 [2019-09-12 17:56:04,101] {{scheduler_job.py:1474}} DEBUG - Sleeping for 0.99 
seconds to prevent excessive logging
 [2019-09-12 17:56:04,126] {{scheduler_job.py:257}} DEBUG - Waiting for 

 [2019-09-12 17:56:04,127] {{scheduler_job.py:257}} DEBUG - Waiting for 

 [2019-09-12 17:56:04,162] {{settings.py:238}} DEBUG - Disposing DB connection 
pool (PID 55)
 [2019-09-12 17:56:04,223] {{settings.py:238}} DEBUG - Disposing DB connection 
pool (PID 58)
 [2019-09-12 17:56:05,095] {{scheduler_job.py:1390}} DEBUG - Starting Loop...
 [2019-09-12 17:56:05,095] {{scheduler_job.py:1401}} DEBUG - Harvesting DAG 
parsing results
 [2019-09-12 17:56:05,097] {{dag_processing.py:637}} DEBUG - Received message 
of type DagParsingStat
 [2019-09-12 17:56:05,098] {{dag_processing.py:637}} DEBUG - Received message 
of type SimpleDag
 [2019-09-12 17:56:05,098] {{dag_processing.py:637}} DEBUG - Received message 
of type SimpleDag
 [2019-09-12 17:56:05,099] {{dag_processing.py:637}} DEBUG - Received message 
of type SimpleDag
 [2019-09-12 17:56:05,099] {{dag_processing.py:637}} DEBUG - Received message 
of type SimpleDag
 [2019-09-12 17:56:05,100] {{dag_processing.py:637}} DEBUG - Received message 
of type DagParsingStat
 [2019-09-12 17:56:05,101] {{dag_processing.py:637}} DEBUG - Received message 
of type DagParsingStat
 [2019-09-12 17:56:05,101] {{scheduler_job.py:1403}} DEBUG - Harvested 4 
SimpleDAGs
 [2019-09-12 17:56:05,128] {{scheduler_job.py:921}} INFO - 5 tasks up for 
execution:
 
 
 
 
 
 [2019-09-12 17:56:05,138] {{scheduler_job.py:953}} INFO - Figuring out tasks 
to run in Pool(name=default_pool) with 128 open slots and 5 task instances 
ready to be queued
 [2019-09-12 17:56:05,139] {{scheduler_job.py:981}} INFO - DAG 
example_subdag_operator has 0/48 running and queued tasks
 [2019-09-12 17:56:05,139] {{scheduler_job.py:981}} INFO - DAG 
latest_only_with_trigger has 0/48 running and queued tasks
 [2019-09-12 17:56:05,139] {{scheduler_job.py:981}} INFO - DAG 
latest_only_with_trigger has 1/48 running and queued tasks
 [2019-09-12 17:56:05,139] {{scheduler_job.py:981}} INFO - DAG 
latest_only_with_trigger has 2/48 running and queued tasks
 [2019-09-12 17:56:05,139] {{scheduler_job.py:981}} INFO - DAG 
latest_only_with_trigger has 3/48 running and queued tasks
 [2019-09-12 17:56:05,139] {{scheduler_job.py:257}} DEBUG -

[jira] [Commented] (AIRFLOW-5447) KubernetesExecutor hangs on task queueing



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928788#comment-16928788
 ] 

Henry Cohen commented on AIRFLOW-5447:
--

```[2019-09-12 17:56:03,034] \{{kubernetes_executor.py:698}} INFO - 
TaskInstance:  found in queued state but was not launched, rescheduling
[2019-09-12 17:56:03,043] \{{scheduler_job.py:1376}} INFO - Resetting orphaned 
tasks for active dag runs
[2019-09-12 17:56:03,085] \{{base_job.py:308}} INFO - Reset the following 30 
TaskInstances:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
[2019-09-12 17:56:03,092] \{{dag_processing.py:545}} INFO - Launched 
DagFileProcessorManager with pid: 35
[2019-09-12 17:56:03,093] \{{scheduler_job.py:1390}} DEBUG - Starting Loop...
[2019-09-12 17:56:03,093] \{{scheduler_job.py:1401}} DEBUG - Harvesting DAG 
parsing results
[2019-09-12 17:56:03,093] \{{scheduler_job.py:1403}} DEBUG - Harvested 0 
SimpleDAGs
[2019-09-12 17:56:03,093] \{{scheduler_job.py:1438}} DEBUG - Heartbeating the 
executor
[2019-09-12 17:56:03,093] \{{base_executor.py:124}} DEBUG - 0 running task 
instances
[2019-09-12 17:56:03,094] \{{base_executor.py:125}} DEBUG - 0 in queue
[2019-09-12 17:56:03,094] \{{base_executor.py:126}} DEBUG - 96 open slots
[2019-09-12 17:56:03,094] \{{base_executor.py:135}} DEBUG - Calling the  sync method
[2019-09-12 17:56:03,100] \{{scheduler_job.py:1459}} DEBUG - Ran scheduling 
loop in 0.01 seconds
[2019-09-12 17:56:03,101] \{{scheduler_job.py:1462}} DEBUG - Sleeping for 1.00 
seconds
[2019-09-12 17:56:03,107] \{{settings.py:54}} INFO - Configured default 
timezone 
[2019-09-12 17:56:03,109] \{{settings.py:327}} DEBUG - Failed to import 
airflow_local_settings.
Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/airflow/settings.py", line 315, 
in import_local_settings
 import airflow_local_settings
ModuleNotFoundError: No module named 'airflow_local_settings'
[2019-09-12 17:56:03,111] \{{logging_config.py:47}} INFO - Successfully 
imported user-defined logging config from log_config.LOGGING_CONFIG
[2019-09-12 17:56:03,120] \{{settings.py:170}} DEBUG - Setting up DB connection 
pool (PID 35)
[2019-09-12 17:56:03,121] \{{settings.py:213}} INFO - settings.configure_orm(): 
Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=35
[2019-09-12 17:56:03,289] \{{settings.py:238}} DEBUG - Disposing DB connection 
pool (PID 45)
[2019-09-12 17:56:03,356] \{{settings.py:238}} DEBUG - Disposing DB connection 
pool (PID 41)
[2019-09-12 17:56:04,101] \{{scheduler_job.py:1474}} DEBUG - Sleeping for 0.99 
seconds to prevent excessive logging
[2019-09-12 17:56:04,126] \{{scheduler_job.py:257}} DEBUG - Waiting for 

[2019-09-12 17:56:04,127] \{{scheduler_job.py:257}} DEBUG - Waiting for 

[2019-09-12 17:56:04,162] \{{settings.py:238}} DEBUG - Disposing DB connection 
pool (PID 55)
[2019-09-12 17:56:04,223] \{{settings.py:238}} DEBUG - Disposing DB connection 
pool (PID 58)
[2019-09-12 17:56:05,095] \{{scheduler_job.py:1390}} DEBUG - Starting Loop...
[2019-09-12 17:56:05,095] \{{scheduler_job.py:1401}} DEBUG - Harvesting DAG 
parsing results
[2019-09-12 17:56:05,097] \{{dag_processing.py:637}} DEBUG - Received message 
of type DagParsingStat
[2019-09-12 17:56:05,098] \{{dag_processing.py:637}} DEBUG - Received message 
of type SimpleDag
[2019-09-12 17:56:05,098] \{{dag_processing.py:637}} DEBUG - Received message 
of type SimpleDag
[2019-09-12 17:56:05,099] \{{dag_processing.py:637}} DEBUG - Received message 
of type SimpleDag
[2019-09-12 17:56:05,099] \{{dag_processing.py:637}} DEBUG - Received message 
of type SimpleDag
[2019-09-12 17:56:05,100] \{{dag_processing.py:637}} DEBUG - Received message 
of type DagParsingStat
[2019-09-12 17:56:05,101] \{{dag_processing.py:637}} DEBUG - Received message 
of type DagParsingStat
[2019-09-12 17:56:05,101] \{{scheduler_job.py:1403}} DEBUG - Harvested 4 
SimpleDAGs
[2019-09-12 17:56:05,128] \{{scheduler_job.py:921}} INFO - 5 tasks up for 
execution:
 
 
 
 
 
[2019-09-12 17:56:05,138] \{{scheduler_job.py:953}} INFO - Figuring out tasks 
to run in Pool(name=default_pool) with 128 open slots and 5 task instances 
ready to be queued
[2019-09-12 17:56:05,139] \{{scheduler_job.py:981}} INFO - DAG 
example_subdag_operator has 0/48 running and queued tasks
[2019-09-12 17:56:05,139] \{{scheduler_job.py:981}} INFO - DAG 
latest_only_with_trigger has 0/48 running and queued tasks
[2019-09-12 17:56:05,139] \{{scheduler_job.py:981}} INFO - DAG 
latest_only_with_trigger has 1/48 running and queued tasks
[2019-09-12 17:56:05,139] \{{scheduler_job.py:981}} INFO - DAG 
latest_only_with_trigger has 2/48 running and queued tasks
[2019-09-12 17:56:05,139] \{{scheduler_job.py:981}} INFO - DAG 
latest_only_with_trigger has 3/48 running and queued tasks
[2019-09-12 17:56:05,139] \{{scheduler_job.py:257}} DEBUG - Waiting for 

[2019-09-12 17:56:05,140] \{{sched

[GitHub] [airflow] mik-laj opened a new pull request #6086: [AIRFLOW-XXX] Display GCP integration in table

mik-laj opened a new pull request #6086: [AIRFLOW-XXX] Display GCP integration 
in table
URL: https://github.com/apache/airflow/pull/6086
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

feluelle commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323864468
 
 

 ##
 File path: airflow/kubernetes/volume.py
 ##
 @@ -25,20 +25,22 @@
 
 
 class Volume(K8SModel):
+"""
+Adds Kubernetes Volume to pod. allows pod to access features like 
ConfigMaps
+and Persistent Volumes
+:param name: the name of the volume mount
+:type name: str
+:param configs: dictionary of any features needed for volume.
+We purposely keep this vague since there are multiple volume types with 
changing
 
 Review comment:
   I think it needs some indention. 2 or 4 spaces?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

feluelle commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323864228
 
 

 ##
 File path: airflow/kubernetes/volume.py
 ##
 @@ -25,20 +25,22 @@
 
 
 class Volume(K8SModel):
+"""
+Adds Kubernetes Volume to pod. allows pod to access features like 
ConfigMaps
+and Persistent Volumes
 
 Review comment:
   ```suggestion
   and Persistent Volumes
   
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

feluelle commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323862535
 
 

 ##
 File path: airflow/kubernetes/volume.py
 ##
 @@ -25,20 +25,22 @@
 
 
 class Volume(K8SModel):
+"""
+Adds Kubernetes Volume to pod. allows pod to access features like 
ConfigMaps
+and Persistent Volumes
 
 Review comment:
   ```suggestion
   and Persistent Volumes
   
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

feluelle commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323862961
 
 

 ##
 File path: airflow/kubernetes/volume.py
 ##
 @@ -25,20 +25,22 @@
 
 
 class Volume(K8SModel):
+"""
+Adds Kubernetes Volume to pod. allows pod to access features like 
ConfigMaps
+and Persistent Volumes
+:param name: the name of the volume mount
+:type name: str
+:param configs: dictionary of any features needed for volume.
+We purposely keep this vague since there are multiple volume types with 
changing
 
 Review comment:
   I think there needs to be some indention.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

feluelle commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323861756
 
 

 ##
 File path: airflow/kubernetes/volume.py
 ##
 @@ -25,20 +25,22 @@
 
 
 class Volume(K8SModel):
+"""
+Adds Kubernetes Volume to pod. allows pod to access features like 
ConfigMaps
 
 Review comment:
   I missed that sorry.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

feluelle commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323862961
 
 

 ##
 File path: airflow/kubernetes/volume.py
 ##
 @@ -25,20 +25,22 @@
 
 
 class Volume(K8SModel):
+"""
+Adds Kubernetes Volume to pod. allows pod to access features like 
ConfigMaps
+and Persistent Volumes
+:param name: the name of the volume mount
+:type name: str
+:param configs: dictionary of any features needed for volume.
+We purposely keep this vague since there are multiple volume types with 
changing
 
 Review comment:
   I think there needs to be some indention.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

feluelle commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323862535
 
 

 ##
 File path: airflow/kubernetes/volume.py
 ##
 @@ -25,20 +25,22 @@
 
 
 class Volume(K8SModel):
+"""
+Adds Kubernetes Volume to pod. allows pod to access features like 
ConfigMaps
+and Persistent Volumes
 
 Review comment:
   ```suggestion
   and Persistent Volumes
   
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

feluelle commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323861756
 
 

 ##
 File path: airflow/kubernetes/volume.py
 ##
 @@ -25,20 +25,22 @@
 
 
 class Volume(K8SModel):
+"""
+Adds Kubernetes Volume to pod. allows pod to access features like 
ConfigMaps
 
 Review comment:
   I missed that sorry.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-4730) Startup-timeout for launching pods on k8s executor/operator should be configurable



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Imberman updated AIRFLOW-4730:
-
Description: 
!Screen Shot 2019-06-04 at 9.04.31 AM.png|width=984,height=203!

Currently users that have affinities for their DAGs are getting failures when 
k8s fails to schedule due to lack of available nodes.

It looks like this might have to do with the fact that the k8s executor uses 
run_pod_async meaning that it attempts it once and then fails on any failure 
from the API. Could probably add logic to read the API section for affinity 
failures

[https://github.com/apache/airflow/blob/05c06b0f6669f677495328c68c2bd05f6d0e69db/airflow/kubernetes/pod_launcher.py#L59]

  was:
!Screen Shot 2019-06-04 at 9.04.31 AM.png!

Currently users that have affinities for their DAGs are getting failures when 
k8s fails to schedule due to lack of available nodes.



It looks like this might have to do with the fact that the k8s executor uses 
run_pod_async meaning that it attempts it once and then fails on any failure 
from the API. Could probably add logic to read the API section for affinity 
failures

https://github.com/apache/airflow/blob/05c06b0f6669f677495328c68c2bd05f6d0e69db/airflow/kubernetes/pod_launcher.py#L59


> Startup-timeout for launching pods on k8s executor/operator should be 
> configurable
> --
>
> Key: AIRFLOW-4730
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4730
> Project: Apache Airflow
>  Issue Type: Task
>  Components: executors
>Affects Versions: 1.10.3
>Reporter: Daniel Imberman
>Priority: Minor
>  Labels: beginner, kubernetes, starter
> Attachments: Screen Shot 2019-06-04 at 9.04.31 AM.png
>
>
> !Screen Shot 2019-06-04 at 9.04.31 AM.png|width=984,height=203!
> Currently users that have affinities for their DAGs are getting failures when 
> k8s fails to schedule due to lack of available nodes.
> It looks like this might have to do with the fact that the k8s executor uses 
> run_pod_async meaning that it attempts it once and then fails on any failure 
> from the API. Could probably add logic to read the API section for affinity 
> failures
> [https://github.com/apache/airflow/blob/05c06b0f6669f677495328c68c2bd05f6d0e69db/airflow/kubernetes/pod_launcher.py#L59]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AIRFLOW-5447) KubernetesExecutor hangs on task queueing

2019-09-12 Thread Kaxil Naik (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928757#comment-16928757
 ] 

Kaxil Naik commented on AIRFLOW-5447:
-

Yes please add some stack trace and we will see at our end if we can replicate 
it.


> KubernetesExecutor hangs on task queueing
> -
>
> Key: AIRFLOW-5447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5447
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: executor-kubernetes
>Affects Versions: 1.10.4, 1.10.5
> Environment: Kubernetes version v1.14.3, Airflow version 1.10.4-1.10.5
>Reporter: Henry Cohen
>Assignee: Daniel Imberman
>Priority: Blocker
>
> Starting in 1.10.4, and continuing in 1.10.5, when using the 
> KubernetesExecutor, with the webserver and scheduler running in the 
> kubernetes cluster, tasks are scheduled, but when added to the task queue, 
> the executor process hangs indefinitely. Based on log messages, it appears to 
> be stuck at this line 
> https://github.com/apache/airflow/blob/v1-10-stable/airflow/contrib/executors/kubernetes_executor.py#L761



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AIRFLOW-5447) KubernetesExecutor hangs on task queueing



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928741#comment-16928741
 ] 

Daniel Imberman commented on AIRFLOW-5447:
--

[~Yuval.Itzchakov] [~HPCohen] could you guys please post either logs and/or a 
breaking DAG? I'm going to look into this.

> KubernetesExecutor hangs on task queueing
> -
>
> Key: AIRFLOW-5447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5447
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: executor-kubernetes
>Affects Versions: 1.10.4, 1.10.5
> Environment: Kubernetes version v1.14.3, Airflow version 1.10.4-1.10.5
>Reporter: Henry Cohen
>Assignee: Daniel Imberman
>Priority: Blocker
>
> Starting in 1.10.4, and continuing in 1.10.5, when using the 
> KubernetesExecutor, with the webserver and scheduler running in the 
> kubernetes cluster, tasks are scheduled, but when added to the task queue, 
> the executor process hangs indefinitely. Based on log messages, it appears to 
> be stuck at this line 
> https://github.com/apache/airflow/blob/v1-10-stable/airflow/contrib/executors/kubernetes_executor.py#L761



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AIRFLOW-5447) KubernetesExecutor hangs on task queueing



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928738#comment-16928738
 ] 

Daniel Imberman commented on AIRFLOW-5447:
--

Oof thank you for bringing this to my attention. cc: [~kaxilnaik] [~ash]

> KubernetesExecutor hangs on task queueing
> -
>
> Key: AIRFLOW-5447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5447
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: executor-kubernetes
>Affects Versions: 1.10.4, 1.10.5
> Environment: Kubernetes version v1.14.3, Airflow version 1.10.4-1.10.5
>Reporter: Henry Cohen
>Assignee: Daniel Imberman
>Priority: Blocker
>
> Starting in 1.10.4, and continuing in 1.10.5, when using the 
> KubernetesExecutor, with the webserver and scheduler running in the 
> kubernetes cluster, tasks are scheduled, but when added to the task queue, 
> the executor process hangs indefinitely. Based on log messages, it appears to 
> be stuck at this line 
> https://github.com/apache/airflow/blob/v1-10-stable/airflow/contrib/executors/kubernetes_executor.py#L761



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [airflow] codecov-io edited a comment on issue #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

codecov-io edited a comment on issue #5786:  [AIRFLOW-5170] [AIRFLOW-5256] 
Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#issuecomment-530201564
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5786?src=pr&el=h1) 
Report
   > Merging 
[#5786](https://codecov.io/gh/apache/airflow/pull/5786?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/19e2f2c7943646d3f7be31f85aafbe6854876d0c?src=pr&el=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `84.8%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5786/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/5786?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #5786  +/-   ##
   =
   + Coverage   80.09%   80.1%   +0.01% 
   =
 Files 606 606  
 Lines   34893   34916  +23 
   =
   + Hits27949   27971  +22 
   - Misses   69446945   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5786?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/5786/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/utils/strings.py](https://codecov.io/gh/apache/airflow/pull/5786/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdHJpbmdzLnB5)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/gcp/example\_dags/example\_bigtable.py](https://codecov.io/gh/apache/airflow/pull/5786/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlndGFibGUucHk=)
 | `0% <ø> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/5786/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/gcp/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/5786/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvbWxlbmdpbmUucHk=)
 | `20.66% <ø> (ø)` | :arrow_up: |
   | 
[airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/5786/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=)
 | `95.34% <ø> (ø)` | :arrow_up: |
   | 
[airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/5786/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=)
 | `84% <ø> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/5786/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/ssh\_hook.py](https://codecov.io/gh/apache/airflow/pull/5786/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NzaF9ob29rLnB5)
 | `87.5% <ø> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/pod.py](https://codecov.io/gh/apache/airflow/pull/5786/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZC5weQ==)
 | `91.66% <ø> (ø)` | :arrow_up: |
   | ... and [26 
more](https://codecov.io/gh/apache/airflow/pull/5786/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5786?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5786?src=pr&el=footer). 
Last update 
[19e2f2c...6f01e2b](https://codecov.io/gh/apache/airflow/pull/5786?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] marengaz edited a comment on issue #6078: [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator

marengaz edited a comment on issue #6078: [AIRFLOW-5459] Use a dynamic tmp 
location in Dataflow operator
URL: https://github.com/apache/airflow/pull/6078#issuecomment-530914266
 
 
   @mik-laj - agree, that would be ideal, but im not sure it is possible in 
this case without a bigger change. see the docstring:
   ```
   Checks whether the file specified by file_name is stored in Google 
Cloud
   Storage (GCS), if so, downloads the file and saves it locally. The 
full
   path of the saved file will be returned. Otherwise the local 
file_name
   will be returned immediately.
   ```
   after this function has been called, it's not possible to know if the file 
is ephemeral or not


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] marengaz commented on issue #6078: [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator

marengaz commented on issue #6078: [AIRFLOW-5459] Use a dynamic tmp location in 
Dataflow operator
URL: https://github.com/apache/airflow/pull/6078#issuecomment-530914266
 
 
   @mik-laj - agree, that would be ideal, but im not sure it is possible in 
this case without a bigger change. see the docstring:
   ```Checks whether the file specified by file_name is stored in Google Cloud
   Storage (GCS), if so, downloads the file and saves it locally. The 
full
   path of the saved file will be returned. Otherwise the local 
file_name
   will be returned immediately.
   ```
   after this function has been called, it's not possible to know if the file 
is ephemeral or not


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

potiuk commented on issue #5786:  [AIRFLOW-5170] [AIRFLOW-5256] Consistent 
licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#issuecomment-530902735
 
 
   Should be all fixed now @feluelle 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

potiuk commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323834134
 
 

 ##
 File path: tests/utils/log/elasticmock/fake_elasticsearch.py
 ##
 @@ -124,6 +135,17 @@ def get(self, index, id, doc_type='_all', params=None):
 
 return result
 
+def find_document(self, doc_type, id, index, result):
 
 Review comment:
   Not sure what was the intention there - I am not the author :). This is a 
client for elastic search and maybe someone would like to use it. I think it's 
out of the scope for this change to change it :).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

potiuk commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323833072
 
 

 ##
 File path: docs/exts/exampleinclude.py
 ##
 @@ -149,15 +157,15 @@ def register_source(app, env, modname):
 code, tags = code_tags
 
 if entry is None or entry[0] != code:
-# print("Registeted", entry)
-
 entry = code, tags, {}, ""
 env._viewcode_modules[modname] = entry  # type: ignore
 
 return True
+# pylint: enable=protected-access
 
 
-def create_node(app, env, relative_path, show_button):
+def create_node(env, relative_path, show_button):
+"""Creates node from module"""
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

potiuk commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323833020
 
 

 ##
 File path: docs/exts/exampleinclude.py
 ##
 @@ -113,27 +116,32 @@ def run(self):
 extra_args["linenostart"] = reader.lineno_start
 
 container_node = nodes.container("", literal_block=True, 
classes=["example-block-wrapper"])
-container_node += example_header(filename=filename)
+container_node += ExampleHeader(filename=filename)
 container_node += retnode
 retnode = container_node
 
 return [retnode]
-except Exception as exc:
+except Exception as exc:  # pylint: disable=broad-except
 return [document.reporter.warning(str(exc), line=self.lineno)]
 
 
+# pylint: disable=protected-access
+# noinspection PyProtectedMember
 def register_source(app, env, modname):
+"""Registers source code"""
 
 Review comment:
   Yep. Fixed all of those.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes

potiuk commented on a change in pull request #5786:  [AIRFLOW-5170] 
[AIRFLOW-5256] Consistent licences for python files and related pylint fixes
URL: https://github.com/apache/airflow/pull/5786#discussion_r323832945
 
 

 ##
 File path: airflow/kubernetes/volume_mount.py
 ##
 @@ -24,26 +24,25 @@
 
 
 class VolumeMount(K8SModel):
-"""Defines Kubernetes Volume Mount"""
-
+"""Initialize a Kubernetes Volume Mount. Used to mount pod level volumes to
+running container.
 
 Review comment:
   Corrected that too.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill

milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using 
scheduler instead of backfill
URL: https://github.com/apache/airflow/pull/5498#issuecomment-530896915
 
 
   I think it is expected that the sub-DAG does not execute when it is paused. 
When we pause/un-pause the parent DAG, its sub-DAGs would be paused/un-paused 
accordingly.
   
   sub-DAG would only execute after its dependencies are met. It should not be 
free-floating like you described. Can you check if we have same issue on latest 
master?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6085: [AIRFLOW-5468] Fix PubSub system test class name

codecov-io commented on issue #6085: [AIRFLOW-5468] Fix PubSub system test 
class name
URL: https://github.com/apache/airflow/pull/6085#issuecomment-530883636
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6085?src=pr&el=h1) 
Report
   > Merging 
[#6085](https://codecov.io/gh/apache/airflow/pull/6085?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/a87f3dfc0c513b05271cba7051c18e4e742ef245?src=pr&el=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6085/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6085?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6085  +/-   ##
   ==
   - Coverage   80.09%   80.08%   -0.01% 
   ==
 Files 606  606  
 Lines   3489334893  
   ==
   - Hits2794627945   -1 
   - Misses   6947 6948   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6085?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6085/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `58.8% <0%> (-0.19%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6085?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6085?src=pr&el=footer). 
Last update 
[a87f3df...7405110](https://codecov.io/gh/apache/airflow/pull/6085?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj merged pull request #6081: [AIRFLOW-5464] Fix GCP Memorystore example DAG

mik-laj merged pull request #6081: [AIRFLOW-5464] Fix GCP Memorystore example 
DAG
URL: https://github.com/apache/airflow/pull/6081
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5464) Fix GCP Memorystore example DAG

2019-09-12 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928659#comment-16928659
 ] 

ASF GitHub Bot commented on AIRFLOW-5464:
-

mik-laj commented on pull request #6081: [AIRFLOW-5464] Fix GCP Memorystore 
example DAG
URL: https://github.com/apache/airflow/pull/6081
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix GCP Memorystore example DAG
> ---
>
> Key: AIRFLOW-5464
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5464
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Tomasz Urbaszek
>Priority: Major
> Fix For: 1.10.6
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Closed] (AIRFLOW-5464) Fix GCP Memorystore example DAG

2019-09-12 Thread Kamil Bregula (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Bregula closed AIRFLOW-5464.
--
Fix Version/s: 1.10.6
   Resolution: Fixed

> Fix GCP Memorystore example DAG
> ---
>
> Key: AIRFLOW-5464
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5464
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Tomasz Urbaszek
>Priority: Major
> Fix For: 1.10.6
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AIRFLOW-5464) Fix GCP Memorystore example DAG



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928660#comment-16928660
 ] 

ASF subversion and git services commented on AIRFLOW-5464:
--

Commit 19e2f2c7943646d3f7be31f85aafbe6854876d0c in airflow's branch 
refs/heads/master from Tomek
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=19e2f2c ]

[AIRFLOW-5464] Fix GCP Memorystore example DAG (#6081)



> Fix GCP Memorystore example DAG
> ---
>
> Key: AIRFLOW-5464
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5464
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Tomasz Urbaszek
>Priority: Major
> Fix For: 1.10.6
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AIRFLOW-5465) Fix deprecated imports in examples

2019-09-12 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928658#comment-16928658
 ] 

ASF subversion and git services commented on AIRFLOW-5465:
--

Commit 9767d5b1a5e9f842edf02cdf0d81c9acd91fada2 in airflow's branch 
refs/heads/master from Tomek
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9767d5b ]

[AIRFLOW-5465] Fix deprecated imports in examples (#6082)



> Fix deprecated imports in examples
> --
>
> Key: AIRFLOW-5465
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5465
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Tomasz Urbaszek
>Priority: Major
> Fix For: 1.10.6
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Closed] (AIRFLOW-5465) Fix deprecated imports in examples

2019-09-12 Thread Kamil Bregula (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Bregula closed AIRFLOW-5465.
--
Fix Version/s: 1.10.6
   Resolution: Fixed

> Fix deprecated imports in examples
> --
>
> Key: AIRFLOW-5465
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5465
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Tomasz Urbaszek
>Priority: Major
> Fix For: 1.10.6
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AIRFLOW-5465) Fix deprecated imports in examples



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928657#comment-16928657
 ] 

ASF GitHub Bot commented on AIRFLOW-5465:
-

mik-laj commented on pull request #6082: [AIRFLOW-5465] Fix deprecated imports 
in examples
URL: https://github.com/apache/airflow/pull/6082
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix deprecated imports in examples
> --
>
> Key: AIRFLOW-5465
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5465
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Tomasz Urbaszek
>Priority: Major
> Fix For: 1.10.6
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [airflow] mik-laj merged pull request #6082: [AIRFLOW-5465] Fix deprecated imports in examples

mik-laj merged pull request #6082: [AIRFLOW-5465] Fix deprecated imports in 
examples
URL: https://github.com/apache/airflow/pull/6082
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5468) Fix PubSub system test class name

2019-09-12 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928654#comment-16928654
 ] 

ASF GitHub Bot commented on AIRFLOW-5468:
-

mik-laj commented on pull request #6085: [AIRFLOW-5468] Fix PubSub system test 
class name
URL: https://github.com/apache/airflow/pull/6085
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix PubSub system test class name
> -
>
> Key: AIRFLOW-5468
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5468
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Tomasz Urbaszek
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Closed] (AIRFLOW-5468) Fix PubSub system test class name

2019-09-12 Thread Kamil Bregula (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kamil Bregula closed AIRFLOW-5468.
--
Fix Version/s: 1.10.6
   Resolution: Fixed

> Fix PubSub system test class name
> -
>
> Key: AIRFLOW-5468
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5468
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Tomasz Urbaszek
>Priority: Major
> Fix For: 1.10.6
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (AIRFLOW-5468) Fix PubSub system test class name



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928655#comment-16928655
 ] 

ASF subversion and git services commented on AIRFLOW-5468:
--

Commit 379f440bd6e71b814794b50ab14c3fe5be0e4095 in airflow's branch 
refs/heads/master from Tomek
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=379f440 ]

[AIRFLOW-5468] Fix PubSub system test class name (#6085)



> Fix PubSub system test class name
> -
>
> Key: AIRFLOW-5468
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5468
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Tomasz Urbaszek
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [airflow] mik-laj merged pull request #6085: [AIRFLOW-5468] Fix PubSub system test class name

mik-laj merged pull request #6085: [AIRFLOW-5468] Fix PubSub system test class 
name
URL: https://github.com/apache/airflow/pull/6085
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6081: [AIRFLOW-5464] Fix GCP Memorystore example DAG

codecov-io commented on issue #6081: [AIRFLOW-5464] Fix GCP Memorystore example 
DAG
URL: https://github.com/apache/airflow/pull/6081#issuecomment-530872296
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6081?src=pr&el=h1) 
Report
   > Merging 
[#6081](https://codecov.io/gh/apache/airflow/pull/6081?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/a87f3dfc0c513b05271cba7051c18e4e742ef245?src=pr&el=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6081/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6081?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6081  +/-   ##
   ==
   + Coverage   80.09%   80.09%   +<.01% 
   ==
 Files 606  606  
 Lines   3489334893  
   ==
   + Hits2794627949   +3 
   + Misses   6947 6944   -3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6081?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...flow/gcp/example\_dags/example\_cloud\_memorystore.py](https://codecov.io/gh/apache/airflow/pull/6081/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfY2xvdWRfbWVtb3J5c3RvcmUucHk=)
 | `0% <0%> (ø)` | :arrow_up: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6081/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.72% <0%> (+0.5%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6081?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6081?src=pr&el=footer). 
Last update 
[a87f3df...2c3959e](https://codecov.io/gh/apache/airflow/pull/6081?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6084: [AIRFLOW-5449] Users can now set existing variable values to None

codecov-io commented on issue #6084: [AIRFLOW-5449] Users can now set existing 
variable values to None
URL: https://github.com/apache/airflow/pull/6084#issuecomment-530872083
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6084?src=pr&el=h1) 
Report
   > Merging 
[#6084](https://codecov.io/gh/apache/airflow/pull/6084?src=pr&el=desc) into 
[v1-10-stable](https://codecov.io/gh/apache/airflow/commit/c026f642f974f8adf2d00c9a5202e26c118721be?src=pr&el=desc)
 will **decrease** coverage by `0.11%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6084/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6084?src=pr&el=tree)
   
   ```diff
   @@   Coverage Diff@@
   ##   v1-10-stable#6084  +/-   ##
   
   - Coverage 78.85%   78.74%   -0.12% 
   
 Files   509  509  
 Lines 3425434256   +2 
   
   - Hits  2701226974  -38 
   - Misses 7242 7282  +40
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6084?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/variable.py](https://codecov.io/gh/apache/airflow/pull/6084/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdmFyaWFibGUucHk=)
 | `94.87% <100%> (+0.13%)` | :arrow_up: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6084/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/utils/log/wasb\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6084/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvd2FzYl90YXNrX2hhbmRsZXIucHk=)
 | `32.87% <0%> (-9.59%)` | :arrow_down: |
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6084/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `88.07% <0%> (-3.67%)` | :arrow_down: |
   | 
[airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/6084/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5)
 | `94.64% <0%> (-1.79%)` | :arrow_down: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6084/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `87.2% <0%> (-1.6%)` | :arrow_down: |
   | 
[airflow/utils/log/s3\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6084/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvczNfdGFza19oYW5kbGVyLnB5)
 | `98.5% <0%> (-1.5%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6084/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `78.88% <0%> (-1.12%)` | :arrow_down: |
   | 
[airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6084/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=)
 | `72.06% <0%> (-0.3%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6084/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `92.9% <0%> (-0.17%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6084?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6084?src=pr&el=footer). 
Last update 
[c026f64...9814213](https://codecov.io/gh/apache/airflow/pull/6084?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-5467) Tasks in the subdag not cleared when dag_id contains "." and starts after retry



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Description: Tasks in the subdag not cleared when dag_id contains "." and 
starts after retry  (was: Tasks in the subdag are not cleared only when 
SubDagOperator contains "*."* and starts after retry)

> Tasks in the subdag not cleared when dag_id contains "." and starts after 
> retry
> ---
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag not cleared when dag_id contains "." and starts after 
> retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks in the subdag not cleared when dag_id contains "." and starts after retry



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Summary: Tasks in the subdag not cleared when dag_id contains "." and 
starts after retry  (was: Tasks in the subdag not cleared when dag_id contains 
dots and starts after retry)

> Tasks in the subdag not cleared when dag_id contains "." and starts after 
> retry
> ---
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared only when SubDagOperator contains "*."* 
> and starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks in the subdag not cleared when dag_id contains dots and starts after retry



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Description: Tasks in the subdag are not cleared only when SubDagOperator 
contains "*."* and starts after retry  (was: Tasks in the subdag are not 
cleared only when SubDagOperator contains \_dots\_ and starts after retry)

> Tasks in the subdag not cleared when dag_id contains dots and starts after 
> retry
> 
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared only when SubDagOperator contains "*."* 
> and starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks in the subdag not cleared when dag_id contains dots and starts after retry



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Summary: Tasks in the subdag not cleared when dag_id contains dots and 
starts after retry  (was: Tasks in the subdag not cleared when dag_id contains 
_dots_ and starts after retry)

> Tasks in the subdag not cleared when dag_id contains dots and starts after 
> retry
> 
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared only when SubDagOperator contains 
> \_dots\_ and starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks in the subdag not cleared when dag_id contains _dots_ and starts after retry



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Summary: Tasks in the subdag not cleared when dag_id contains _dots_ and 
starts after retry  (was: Tasks in the subdag are not cleared when 
SubDagOperator starts after retry)

> Tasks in the subdag not cleared when dag_id contains _dots_ and starts after 
> retry
> --
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared only when SubDagOperator contains 
> \_dots\_ and starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [airflow] nuclearpinguin opened a new pull request #6085: [AIRFLOW-5468] Fix PubSub system test class name

nuclearpinguin opened a new pull request #6085: [AIRFLOW-5468] Fix PubSub 
system test class name
URL: https://github.com/apache/airflow/pull/6085
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-5468
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5468) Fix PubSub system test class name



[ 
https://issues.apache.org/jira/browse/AIRFLOW-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928609#comment-16928609
 ] 

ASF GitHub Bot commented on AIRFLOW-5468:
-

nuclearpinguin commented on pull request #6085: [AIRFLOW-5468] Fix PubSub 
system test class name
URL: https://github.com/apache/airflow/pull/6085
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-5468
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix PubSub system test class name
> -
>
> Key: AIRFLOW-5468
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5468
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Affects Versions: 1.10.5
>Reporter: Tomasz Urbaszek
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (AIRFLOW-5468) Fix PubSub system test class name

2019-09-12 Thread Tomasz Urbaszek (Jira)

Tomasz Urbaszek created AIRFLOW-5468:


 Summary: Fix PubSub system test class name
 Key: AIRFLOW-5468
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5468
 Project: Apache Airflow
  Issue Type: Bug
  Components: gcp
Affects Versions: 1.10.5
Reporter: Tomasz Urbaszek






--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks in the subdag are not cleared when SubDagOperator starts after retry



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Description: Tasks in the subdag are not cleared only when SubDagOperator 
contains \_dots\_ and starts after retry  (was: Tasks in the subdag are not 
cleared only when SubDagOperator contains _dots_ and starts after retry)

> Tasks in the subdag are not cleared when SubDagOperator starts after retry
> --
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared only when SubDagOperator contains 
> \_dots\_ and starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks in the subdag are not cleared when SubDagOperator starts after retry



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Description: Tasks in the subdag are not cleared only when SubDagOperator 
contains *_dots_ *and starts after retry  (was: Tasks in the subdag are not 
cleared only when SubDagOperator contains _dots_ and starts after retry)

> Tasks in the subdag are not cleared when SubDagOperator starts after retry
> --
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared only when SubDagOperator contains *_dots_ 
> *and starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks in the subdag are not cleared when SubDagOperator starts after retry



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Description: Tasks in the subdag are not cleared only when SubDagOperator 
contains _dots_ and starts after retry  (was: Tasks in the subdag are not 
cleared when SubDagOperator starts after retry)

> Tasks in the subdag are not cleared when SubDagOperator starts after retry
> --
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared only when SubDagOperator contains _dots_ 
> and starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks in the subdag are not cleared when SubDagOperator starts after retry



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Description: Tasks in the subdag are not cleared only when SubDagOperator 
contains _dots_ and starts after retry  (was: Tasks in the subdag are not 
cleared only when SubDagOperator contains *\_dots\_*and starts after retry)

> Tasks in the subdag are not cleared when SubDagOperator starts after retry
> --
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared only when SubDagOperator contains _dots_ 
> and starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks in the subdag are not cleared when SubDagOperator starts after retry



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Description: Tasks in the subdag are not cleared only when SubDagOperator 
contains *\_dots\_*and starts after retry  (was: Tasks in the subdag are not 
cleared only when SubDagOperator contains *_dots_ *and starts after retry)

> Tasks in the subdag are not cleared when SubDagOperator starts after retry
> --
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared only when SubDagOperator contains 
> *\_dots\_*and starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks are not cleared in the subdag with the name that contains the _dot_



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Affects Version/s: (was: 1.10.5)
   1.10.2

> Tasks are not cleared in the subdag with the name that contains the _dot_
> -
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared when SubDagOperator starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks in the subdag are not cleared when SubDagOperator starts after retry



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Summary: Tasks in the subdag are not cleared when SubDagOperator starts 
after retry  (was: Tasks are not cleared in the subdag with the name that 
contains the _dot_)

> Tasks in the subdag are not cleared when SubDagOperator starts after retry
> --
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.2
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared when SubDagOperator starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks are not cleared in the subdag with the name that contains the _dot_



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Description: Tasks in the subdag are not cleared when SubDagOperator starts 
after retry  (was: Tasks are not cleared in the subdag with the name that 
contains the _dot_)

> Tasks are not cleared in the subdag with the name that contains the _dot_
> -
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.5
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks in the subdag are not cleared when SubDagOperator starts after retry



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (AIRFLOW-5467) Tasks are not cleared in the subdag with the name that contains the _dot_



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Kucherov updated AIRFLOW-5467:
-
Priority: Minor  (was: Major)

> Tasks are not cleared in the subdag with the name that contains the _dot_
> -
>
> Key: AIRFLOW-5467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5467
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, models
>Affects Versions: 1.10.5
>Reporter: Andrey Kucherov
>Assignee: Andrey Kucherov
>Priority: Minor
>
> Tasks are not cleared in the subdag with the name that contains the _dot_



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (AIRFLOW-5467) Tasks are not cleared in the subdag with the name that contains the _dot_