[GitHub] [airflow] codecov-io commented on issue #6203: [AIRFLOW-XXX] Add more ASF transfer operators

[GitBox]

codecov-io commented on issue #6203: [AIRFLOW-XXX] Add more ASF transfer 
operators
URL: https://github.com/apache/airflow/pull/6203#issuecomment-536235603
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6203?src=pr&el=h1) 
Report
   > Merging 
[#6203](https://codecov.io/gh/apache/airflow/pull/6203?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6203/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6203?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6203  +/-   ##
   ==
   - Coverage   80.02%   80.02%   -0.01% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815028149   -1 
   - Misses   7026 7027   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6203?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6203/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.72% <0%> (-0.18%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6203?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6203?src=pr&el=footer). 
Last update 
[2350b2f...4260b18](https://codecov.io/gh/apache/airflow/pull/6203?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329331524
 
 

 ##
 File path: airflow/sensors/base_sensor_operator.py
 ##
 @@ -121,7 +122,17 @@ def execute(self, context: Dict) -> None:
 raise AirflowRescheduleException(reschedule_date)
 else:
 sleep(self.poke_interval)
-self.log.info("Success criteria met. Exiting.")
+
+if isinstance(self, BaseAsyncOperator):
 
 Review comment:
   Oh. Isn't it enough to run the code after the loop in the execute method?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329331460
 
 

 ##
 File path: airflow/models/base_async_operator.py
 ##
 @@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must define a `submit_request` to fire a request for a
+long running operation with a method and then executes a `poke` method
+executing at a time interval and succeed when a criteria is met and fail
+if and when they time out. They are effctively an opinionated way use
+combine an Operator and a Sensor in order to kick off a long running
+process without blocking a worker slot while waiting for the long running
+process to complete by leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+:type mode: str
+"""
+ui_color = '#9933ff'  # type: str
+valid_modes = ['poke', 'reschedule']  # type: Iterable[str]
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+def submit_request(self, context) -> string:
+"""
+This method should kick off a long running operation.
+This method should return the ID for the long running operation used
+for polling
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
 
 Review comment:
   ```suggestion
   Refer to :meth:`get_template_context` for more context.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329331460
 
 

 ##
 File path: airflow/models/base_async_operator.py
 ##
 @@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must define a `submit_request` to fire a request for a
+long running operation with a method and then executes a `poke` method
+executing at a time interval and succeed when a criteria is met and fail
+if and when they time out. They are effctively an opinionated way use
+combine an Operator and a Sensor in order to kick off a long running
+process without blocking a worker slot while waiting for the long running
+process to complete by leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+:type mode: str
+"""
+ui_color = '#9933ff'  # type: str
+valid_modes = ['poke', 'reschedule']  # type: Iterable[str]
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+def submit_request(self, context) -> string:
+"""
+This method should kick off a long running operation.
+This method should return the ID for the long running operation used
+for polling
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
 
 Review comment:
   ```suggestion
   Refer to :meth:`get_template_context` for more context.
   ```
   This probably still will not be completely correct and you need to add some 
prefix to the name.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329331454
 
 

 ##
 File path: airflow/models/base_async_operator.py
 ##
 @@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must define a `submit_request` to fire a request for a
+long running operation with a method and then executes a `poke` method
+executing at a time interval and succeed when a criteria is met and fail
+if and when they time out. They are effctively an opinionated way use
+combine an Operator and a Sensor in order to kick off a long running
+process without blocking a worker slot while waiting for the long running
+process to complete by leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+:type mode: str
+"""
+ui_color = '#9933ff'  # type: str
+valid_modes = ['poke', 'reschedule']  # type: Iterable[str]
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+def submit_request(self, context) -> string:
 
 Review comment:
   ```suggestion
   def submit_request(self, context) -> IT:
   ```
   I imagine that the identifier will be a different type, e.g. a number, or an 
array with several identifiers. What do you think about using [generic 
types](https://docs.python.org/3.5/library/typing.html#typing.Generic)?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329331081
 
 

 ##
 File path: airflow/models/base_async_operator.py
 ##
 @@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must define a `submit_request` to fire a request for a
+long running operation with a method and then executes a `poke` method
+executing at a time interval and succeed when a criteria is met and fail
+if and when they time out. They are effctively an opinionated way use
+combine an Operator and a Sensor in order to kick off a long running
+process without blocking a worker slot while waiting for the long running
+process to complete by leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+:type mode: str
+"""
+ui_color = '#9933ff'  # type: str
+valid_modes = ['poke', 'reschedule']  # type: Iterable[str]
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+def submit_request(self, context) -> string:
+"""
+This method should kick off a long running operation.
+This method should return the ID for the long running operation used
+for polling
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+
+:returns: a resource_id for the long running operation.
+:rtype: str
+"""
+raise AirflowException('Async Operators must define a `submit_request` 
method.')
+
+def process_result(self, context):
+"""
+This method can optionally be overriden to process the result of a 
long running operation.
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+"""
+self.log.info('Got result of {}. Done.'.format(
+  self.get_external_resource_id(context))
+
+def pre_execute(self, context) -> None:
+"""
+Check if we have the XCOM_EXTERNAL_RESOURCE_ID_KEY
+for this task and call submit_request if it is missing.
+"""
+if not self.get_external_resource_id(context):
 
 Review comment:
   I think that we should not rely on this content. I can imagine that there 
may be operators that do not operate on any identifier.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329331145
 
 

 ##
 File path: airflow/models/base_async_operator.py
 ##
 @@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must define a `submit_request` to fire a request for a
+long running operation with a method and then executes a `poke` method
+executing at a time interval and succeed when a criteria is met and fail
+if and when they time out. They are effctively an opinionated way use
+combine an Operator and a Sensor in order to kick off a long running
+process without blocking a worker slot while waiting for the long running
+process to complete by leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+:type mode: str
+"""
+ui_color = '#9933ff'  # type: str
+valid_modes = ['poke', 'reschedule']  # type: Iterable[str]
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+def submit_request(self, context) -> string:
+"""
+This method should kick off a long running operation.
+This method should return the ID for the long running operation used
+for polling
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+
+:returns: a resource_id for the long running operation.
+:rtype: str
+"""
+raise AirflowException('Async Operators must define a `submit_request` 
method.')
+
+def process_result(self, context):
+"""
+This method can optionally be overriden to process the result of a 
long running operation.
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+"""
+self.log.info('Got result of {}. Done.'.format(
+  self.get_external_resource_id(context))
+
+def pre_execute(self, context) -> None:
 
 Review comment:
   Executing code in pre_execute is dangerous because you can jam the thread.  
The code executed in the execute method has a timeout.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329331123
 
 

 ##
 File path: airflow/models/base_async_operator.py
 ##
 @@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must define a `submit_request` to fire a request for a
+long running operation with a method and then executes a `poke` method
+executing at a time interval and succeed when a criteria is met and fail
+if and when they time out. They are effctively an opinionated way use
+combine an Operator and a Sensor in order to kick off a long running
+process without blocking a worker slot while waiting for the long running
+process to complete by leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+:type mode: str
+"""
+ui_color = '#9933ff'  # type: str
+valid_modes = ['poke', 'reschedule']  # type: Iterable[str]
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+def submit_request(self, context) -> string:
+"""
+This method should kick off a long running operation.
+This method should return the ID for the long running operation used
+for polling
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+
+:returns: a resource_id for the long running operation.
+:rtype: str
+"""
+raise AirflowException('Async Operators must define a `submit_request` 
method.')
+
+def process_result(self, context):
+"""
+This method can optionally be overriden to process the result of a 
long running operation.
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+"""
+self.log.info('Got result of {}. Done.'.format(
+  self.get_external_resource_id(context))
+
+def pre_execute(self, context) -> None:
+"""
+Check if we have the XCOM_EXTERNAL_RESOURCE_ID_KEY
+for this task and call submit_request if it is missing.
+"""
+if not self.get_external_resource_id(context):
+resource_id = submit_request(self, context)
 
 Review comment:
   ```suggestion
   resource_id = self.submit_request(self, context)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329331113
 
 

 ##
 File path: airflow/models/base_async_operator.py
 ##
 @@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must define a `submit_request` to fire a request for a
+long running operation with a method and then executes a `poke` method
+executing at a time interval and succeed when a criteria is met and fail
+if and when they time out. They are effctively an opinionated way use
+combine an Operator and a Sensor in order to kick off a long running
+process without blocking a worker slot while waiting for the long running
+process to complete by leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+:type mode: str
+"""
+ui_color = '#9933ff'  # type: str
+valid_modes = ['poke', 'reschedule']  # type: Iterable[str]
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+def submit_request(self, context) -> string:
+"""
+This method should kick off a long running operation.
+This method should return the ID for the long running operation used
+for polling
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+
+:returns: a resource_id for the long running operation.
+:rtype: str
+"""
+raise AirflowException('Async Operators must define a `submit_request` 
method.')
+
+def process_result(self, context):
+"""
+This method can optionally be overriden to process the result of a 
long running operation.
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+"""
+self.log.info('Got result of {}. Done.'.format(
+  self.get_external_resource_id(context))
+
+def pre_execute(self, context) -> None:
 
 Review comment:
   Is there a reason why this can't be done in execute?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329331081
 
 

 ##
 File path: airflow/models/base_async_operator.py
 ##
 @@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must define a `submit_request` to fire a request for a
+long running operation with a method and then executes a `poke` method
+executing at a time interval and succeed when a criteria is met and fail
+if and when they time out. They are effctively an opinionated way use
+combine an Operator and a Sensor in order to kick off a long running
+process without blocking a worker slot while waiting for the long running
+process to complete by leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+:type mode: str
+"""
+ui_color = '#9933ff'  # type: str
+valid_modes = ['poke', 'reschedule']  # type: Iterable[str]
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+def submit_request(self, context) -> string:
+"""
+This method should kick off a long running operation.
+This method should return the ID for the long running operation used
+for polling
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+
+:returns: a resource_id for the long running operation.
+:rtype: str
+"""
+raise AirflowException('Async Operators must define a `submit_request` 
method.')
+
+def process_result(self, context):
+"""
+This method can optionally be overriden to process the result of a 
long running operation.
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+"""
+self.log.info('Got result of {}. Done.'.format(
+  self.get_external_resource_id(context))
+
+def pre_execute(self, context) -> None:
+"""
+Check if we have the XCOM_EXTERNAL_RESOURCE_ID_KEY
+for this task and call submit_request if it is missing.
+"""
+if not self.get_external_resource_id(context):
 
 Review comment:
   I think that we should not rely on the content of this field. I can imagine 
that there may be operators that do not operate on any identifier.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329331024
 
 

 ##
 File path: airflow/models/base_async_operator.py
 ##
 @@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must define a `submit_request` to fire a request for a
+long running operation with a method and then executes a `poke` method
+executing at a time interval and succeed when a criteria is met and fail
+if and when they time out. They are effctively an opinionated way use
+combine an Operator and a Sensor in order to kick off a long running
+process without blocking a worker slot while waiting for the long running
+process to complete by leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+:type mode: str
 
 Review comment:
   You pass mode in 56 line, so if someone passes his value to this parameter 
then an exception will be raised.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329330961
 
 

 ##
 File path: airflow/models/base_async_operator.py
 ##
 @@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must define a `submit_request` to fire a request for a
+long running operation with a method and then executes a `poke` method
+executing at a time interval and succeed when a criteria is met and fail
+if and when they time out. They are effctively an opinionated way use
+combine an Operator and a Sensor in order to kick off a long running
+process without blocking a worker slot while waiting for the long running
+process to complete by leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+:type mode: str
+"""
+ui_color = '#9933ff'  # type: str
+valid_modes = ['poke', 'reschedule']  # type: Iterable[str]
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+def submit_request(self, context) -> string:
+"""
+This method should kick off a long running operation.
+This method should return the ID for the long running operation used
+for polling
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+
+:returns: a resource_id for the long running operation.
+:rtype: str
+"""
+raise AirflowException('Async Operators must define a `submit_request` 
method.')
+
+def process_result(self, context):
+"""
+This method can optionally be overriden to process the result of a 
long running operation.
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+"""
+self.log.info('Got result of {}. Done.'.format(
 
 Review comment:
   ```suggestion
   self.log.info('Got result of %s. Done.', 
   ```
   we should avoid formatting string before passing to logger. 
   https://github.com/apache/airflow/pull/5681
   https://github.com/apache/airflow/pull/4804


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on a change in pull request #6210: [AIRFLOW-5567] [Do not 
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r329330961
 
 

 ##
 File path: airflow/models/base_async_operator.py
 ##
 @@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from functools import wraps
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must define a `submit_request` to fire a request for a
+long running operation with a method and then executes a `poke` method
+executing at a time interval and succeed when a criteria is met and fail
+if and when they time out. They are effctively an opinionated way use
+combine an Operator and a Sensor in order to kick off a long running
+process without blocking a worker slot while waiting for the long running
+process to complete by leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+:type mode: str
+"""
+ui_color = '#9933ff'  # type: str
+valid_modes = ['poke', 'reschedule']  # type: Iterable[str]
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+def submit_request(self, context) -> string:
+"""
+This method should kick off a long running operation.
+This method should return the ID for the long running operation used
+for polling
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+
+:returns: a resource_id for the long running operation.
+:rtype: str
+"""
+raise AirflowException('Async Operators must define a `submit_request` 
method.')
+
+def process_result(self, context):
+"""
+This method can optionally be overriden to process the result of a 
long running operation.
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+"""
+self.log.info('Got result of {}. Done.'.format(
 
 Review comment:
   ```suggestion
   self.log.info('Got result of %s. Done.', 
   ```
   we should avoid formatting string before passing to logger. 
   Refernece
   https://github.com/apache/airflow/pull/5681
   https://github.com/apache/airflow/pull/4804


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6209: [AIRFLOW-XXX] Add service transfer operators

[GitBox]

codecov-io edited a comment on issue #6209: [AIRFLOW-XXX] Add service transfer 
operators
URL: https://github.com/apache/airflow/pull/6209#issuecomment-536229920
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=h1) 
Report
   > Merging 
[#6209](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/bfe0ace2808a2a8934fbf578927bde81fc33420e?src=pr&el=desc)
 will **decrease** coverage by `0.6%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6209/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6209  +/-   ##
   ==
   - Coverage   80.02%   79.41%   -0.61% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815027936 -214 
   - Misses   7026 7240 +214
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `40.74% <0%> (-35.56%)` | :arrow_down: |
   | 
[airflow/utils/log/wasb\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvd2FzYl90YXNrX2hhbmRsZXIucHk=)
 | `32.87% <0%> (-9.59%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `84.74% <0%> (-8.48%)` | :arrow_down: |
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `88.07% <0%> (-3.67%)` | :arrow_down: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `84.74% <0%> (-3.39%)` | :arrow_down: |
   | 
[airflow/executors/base\_executor.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvYmFzZV9leGVjdXRvci5weQ==)
 | `92.75% <0%> (-2.9%)` | :arrow_down: |
   | ... and [8 
more](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=footer). 
Last update 
[bfe0ace...5f90b7e](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6209: [AIRFLOW-XXX] Add service transfer operators

[GitBox]

codecov-io edited a comment on issue #6209: [AIRFLOW-XXX] Add service transfer 
operators
URL: https://github.com/apache/airflow/pull/6209#issuecomment-536229920
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=h1) 
Report
   > Merging 
[#6209](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/bfe0ace2808a2a8934fbf578927bde81fc33420e?src=pr&el=desc)
 will **decrease** coverage by `0.07%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6209/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6209  +/-   ##
   ==
   - Coverage   80.02%   79.94%   -0.08% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815028122  -28 
   - Misses   7026 7054  +28
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5)
 | `94.73% <0%> (-1.76%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `91.52% <0%> (-1.7%)` | :arrow_down: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `86.44% <0%> (-1.7%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.24% <0%> (-0.51%)` | :arrow_down: |
   | 
[airflow/contrib/operators/ssh\_operator.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zc2hfb3BlcmF0b3IucHk=)
 | `83.75% <0%> (+1.25%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=footer). 
Last update 
[bfe0ace...5f90b7e](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6209: [AIRFLOW-XXX] Add service transfer operators

[GitBox]

codecov-io commented on issue #6209: [AIRFLOW-XXX] Add service transfer 
operators
URL: https://github.com/apache/airflow/pull/6209#issuecomment-536229920
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=h1) 
Report
   > Merging 
[#6209](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/bfe0ace2808a2a8934fbf578927bde81fc33420e?src=pr&el=desc)
 will **decrease** coverage by `0.6%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6209/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6209  +/-   ##
   ==
   - Coverage   80.02%   79.41%   -0.61% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815027936 -214 
   - Misses   7026 7240 +214
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `40.74% <0%> (-35.56%)` | :arrow_down: |
   | 
[airflow/utils/log/wasb\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvd2FzYl90YXNrX2hhbmRsZXIucHk=)
 | `32.87% <0%> (-9.59%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `84.74% <0%> (-8.48%)` | :arrow_down: |
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `88.07% <0%> (-3.67%)` | :arrow_down: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `84.74% <0%> (-3.39%)` | :arrow_down: |
   | 
[airflow/executors/base\_executor.py](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvYmFzZV9leGVjdXRvci5weQ==)
 | `92.75% <0%> (-2.9%)` | :arrow_down: |
   | ... and [8 
more](https://codecov.io/gh/apache/airflow/pull/6209/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=footer). 
Last update 
[bfe0ace...5f90b7e](https://codecov.io/gh/apache/airflow/pull/6209?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6208: [AIRFLOW-XXX] Add protocol transfer operators

[GitBox]

codecov-io commented on issue #6208: [AIRFLOW-XXX] Add protocol transfer 
operators
URL: https://github.com/apache/airflow/pull/6208#issuecomment-536228452
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6208?src=pr&el=h1) 
Report
   > Merging 
[#6208](https://codecov.io/gh/apache/airflow/pull/6208?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/bfe0ace2808a2a8934fbf578927bde81fc33420e?src=pr&el=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6208/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6208?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6208  +/-   ##
   ==
   - Coverage   80.02%   80.01%   -0.01% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815028147   -3 
   - Misses   7026 7029   +3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6208?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6208/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.24% <0%> (-0.51%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6208?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6208?src=pr&el=footer). 
Last update 
[bfe0ace...5a50d20](https://codecov.io/gh/apache/airflow/pull/6208?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6207: [AIRFLOW-XXX] Add software transfer operators

[GitBox]

codecov-io commented on issue #6207: [AIRFLOW-XXX] Add software transfer 
operators
URL: https://github.com/apache/airflow/pull/6207#issuecomment-536227052
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=h1) 
Report
   > Merging 
[#6207](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **decrease** coverage by `0.72%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6207/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #6207  +/-   ##
   =
   - Coverage   80.02%   79.3%   -0.73% 
   =
 Files 610 610  
 Lines   35176   35176  
   =
   - Hits28150   27896 -254 
   - Misses   70267280 +254
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==)
 | `47.61% <0%> (-52.39%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | 
[airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=)
 | `72.72% <0%> (-20.46%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `77.96% <0%> (-15.26%)` | :arrow_down: |
   | ... and [8 
more](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=footer). 
Last update 
[2350b2f...8fc6af4](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6207: [AIRFLOW-XXX] Add software transfer operators

[GitBox]

codecov-io edited a comment on issue #6207: [AIRFLOW-XXX] Add software transfer 
operators
URL: https://github.com/apache/airflow/pull/6207#issuecomment-536227052
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=h1) 
Report
   > Merging 
[#6207](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **decrease** coverage by `0.46%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6207/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6207  +/-   ##
   ==
   - Coverage   80.02%   79.55%   -0.47% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815027986 -164 
   - Misses   7026 7190 +164
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==)
 | `47.61% <0%> (-52.39%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | 
[airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=)
 | `72.72% <0%> (-20.46%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `86.44% <0%> (-6.78%)` | :arrow_down: |
   | 
[airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=)
 | `63.26% <0%> (-4.09%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `54.13% <0%> (-2.76%)` | :arrow_down: |
   | ... and [2 
more](https://codecov.io/gh/apache/airflow/pull/6207/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=footer). 
Last update 
[2350b2f...8fc6af4](https://codecov.io/gh/apache/airflow/pull/6207?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5569) Improve Pod Launcher Logging

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940160#comment-16940160
 ] 

ASF GitHub Bot commented on AIRFLOW-5569:
-

mik-laj commented on pull request #6211: [AIRFLOW-5569] Improve Pod Launcher 
Logging
URL: https://github.com/apache/airflow/pull/6211
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Improve Pod Launcher Logging
> 
>
> Key: AIRFLOW-5569
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5569
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: executor-kubernetes
>Affects Versions: 1.10.5
>Reporter: Kamil Bregula
>Assignee: Daniel Imberman
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj opened a new pull request #6211: [AIRFLOW-5569] Improve Pod Launcher Logging

[GitBox]

mik-laj opened a new pull request #6211: [AIRFLOW-5569] Improve Pod Launcher 
Logging
URL: https://github.com/apache/airflow/pull/6211
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-5569) Improve Pod Launcher Logging

[Kamil Bregula (Jira)]

Kamil Bregula created AIRFLOW-5569:
--

 Summary: Improve Pod Launcher Logging
 Key: AIRFLOW-5569
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5569
 Project: Apache Airflow
  Issue Type: Improvement
  Components: executor-kubernetes
Affects Versions: 1.10.5
Reporter: Kamil Bregula
Assignee: Daniel Imberman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-5568) Add Hook / Operators for GCP Healthcare API

[Jacob Ferriero (Jira)]

Jacob Ferriero created AIRFLOW-5568:
---

 Summary: Add Hook / Operators for GCP Healthcare API
 Key: AIRFLOW-5568
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5568
 Project: Apache Airflow
  Issue Type: New Feature
  Components: hooks, operators
Affects Versions: 1.10.5
Reporter: Jacob Ferriero


It'd be useful to have a hook for the healthcare api

and some operators / sensor for the long running operations 
(https://cloud.google.com/healthcare/docs/how-tos/long-running-operations)
 * import / export of various formats
 * deidentification of datasets

 [https://cloud.google.com/healthcare/docs/apis]

 

Note this would be a good candidate to illustrate some sort of AysncOperator 
described in AIRFLOW-5567



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io commented on issue #6206: [AIRFLOW-XXX] Add more GCP transfer operators

[GitBox]

codecov-io commented on issue #6206: [AIRFLOW-XXX] Add more GCP transfer 
operators
URL: https://github.com/apache/airflow/pull/6206#issuecomment-536224868
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6206?src=pr&el=h1) 
Report
   > Merging 
[#6206](https://codecov.io/gh/apache/airflow/pull/6206?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6206/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6206?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6206  +/-   ##
   ==
   - Coverage   80.02%   80.01%   -0.02% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815028146   -4 
   - Misses   7026 7030   +4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6206?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6206/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.24% <0%> (-0.51%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6206/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.72% <0%> (-0.18%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6206?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6206?src=pr&el=footer). 
Last update 
[2350b2f...3c583ce](https://codecov.io/gh/apache/airflow/pull/6206?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5342) Can't run initdb using Microsoft SQL server

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940157#comment-16940157
 ] 

ASF subversion and git services commented on AIRFLOW-5342:
--

Commit f9981670c14da842b8ce351dfc90b6ad271b42ca in airflow's branch 
refs/heads/v1-10-test from Morten Post
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f998167 ]

[AIRFLOW-5342] Fix MSSQL breaking task_instance db migration

MSSQL does not allow altering columns to NOT NULL when the column
is used in an index. Therefore we drop the ti_pool index and recreate
it after modifying the column.

Co-authored-by: mattinbits <3765307+mattinb...@users.noreply.github.com>
Co-authored-by: sirVir 
(cherry picked from commit f1674e30d36c617e95e3b3762739197932726f4e)


> Can't run initdb using Microsoft SQL server
> ---
>
> Key: AIRFLOW-5342
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5342
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration, core, database
>Affects Versions: 1.10.4
>Reporter: Adam Trump
>Assignee: Morten Post
>Priority: Critical
> Fix For: 1.10.6
>
>
> I'm using Microsoft SQL server 2016 as metadata db (for the 
> sql_alchemy_conn), and pymssql as the Python driver.
> When running airflow initdb, airflow creates a table named `TaskInstance` 
> with column `pool` that is initially Nullable (when the table is created).
> Later, an index named `ti_pool` on a few columns, including `pool` column in 
> table `TaskInstance`.
> Then, airflow will try to alter table `TaskInstance` and change the column 
> `pool` to `NOT NULL`.
> This does not work on Microsoft SQL Server since a column with an index 
> defined on it cannot be changed, unless the index is deleted before.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj commented on issue #6210: [AIRFLOW-5567] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on issue #6210: [AIRFLOW-5567] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#issuecomment-536223775
 
 
   I think it's worth introducing a standardized way of passing resource ID. 
This will also be very useful in [extra 
links](https://airflow.readthedocs.io/en/latest/howto/define_extra_link.html).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jaketf commented on issue #6210: [AIRFLOW-5567] prototype BaseAsyncOperator

[GitBox]

jaketf commented on issue #6210: [AIRFLOW-5567] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#issuecomment-536223550
 
 
   @mik-laj this is mostly a discussion piece of a PR so thanks for joining in!
   I think post execute would be good. IMO you could just use XCom to 
communicate the id (if you need to). Do you think there's value in doing it 
another way?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6205: [AIRFLOW-XXX] Add more AWS transfer operators

[GitBox]

codecov-io commented on issue #6205: [AIRFLOW-XXX] Add more AWS transfer 
operators
URL: https://github.com/apache/airflow/pull/6205#issuecomment-536223416
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=h1) 
Report
   > Merging 
[#6205](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6205/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#6205   +/-   ##
   ===
 Coverage   80.02%   80.02%   
   ===
 Files 610  610   
 Lines   3517635176   
   ===
 Hits2815028150   
 Misses   7026 7026
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=footer). 
Last update 
[2350b2f...2369d42](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6205: [AIRFLOW-XXX] Add more AWS transfer operators

[GitBox]

codecov-io edited a comment on issue #6205: [AIRFLOW-XXX] Add more AWS transfer 
operators
URL: https://github.com/apache/airflow/pull/6205#issuecomment-536223416
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=h1) 
Report
   > Merging 
[#6205](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6205/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#6205   +/-   ##
   ===
 Coverage   80.02%   80.02%   
   ===
 Files 610  610   
 Lines   3517635176   
   ===
 Hits2815028150   
 Misses   7026 7026
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=footer). 
Last update 
[2350b2f...2369d42](https://codecov.io/gh/apache/airflow/pull/6205?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-630) Airflow worker is not working with Celery 4.0.0

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940155#comment-16940155
 ] 

Sergio Kef commented on AIRFLOW-630:


Can we close this?

> Airflow worker is not working with Celery 4.0.0
> ---
>
> Key: AIRFLOW-630
> URL: https://issues.apache.org/jira/browse/AIRFLOW-630
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
>Affects Versions: 1.7.1.2, 1.7.1.3
>Reporter: Hafiz Badrie Lubis
>Priority: Major
>
> Soon as celery version is upgraded to 4.0.0, airflow worker is not working, 
> because loglevel value is None. You can see the detail of error log on this 
> image: http://imgur.com/JHedHeN. 
> Should make loglevel value assignment be more flexible.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj merged pull request #6201: [AIRFLOW-XXX] Improve style - missing commas, redundant dots

[GitBox]

mik-laj merged pull request #6201: [AIRFLOW-XXX] Improve style - missing 
commas, redundant dots
URL: https://github.com/apache/airflow/pull/6201
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj edited a comment on issue #6210: [AIRFLOW-5567] prototype BaseAsyncOperator

[GitBox]

mik-laj edited a comment on issue #6210: [AIRFLOW-5567] prototype 
BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#issuecomment-536221991
 
 
   My team was thinking of creating similar features but we haven't started 
work and it's possible that the plans will change. However, I will gladly help 
in implementing this features in the community.
   
   Some observation that I can add now it is useful that the functionality of 
the code execution **after** the operation is completed.
   For example:
   * Pre-execute: create a report request in Google Marketing Platform
   * Poke: Waiting for status change from processing to completed
   * Post-execute: download the raport to GCS
   
   Code execution only before the operator is insufficient in more complex 
processes.
   
   The current implementation lacks a mechanism for passing the context of 
operations to the next step, e.g. the ID of the created object.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5567) Improved primitive for building Operators that benefit from reschedule mode

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940153#comment-16940153
 ] 

ASF GitHub Bot commented on AIRFLOW-5567:
-

jaketf commented on pull request #6210: [AIRFLOW-5567] prototype 
BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Improved primitive for building Operators that benefit from reschedule mode
> ---
>
> Key: AIRFLOW-5567
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5567
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: models, operators
>Affects Versions: 1.10.5
>Reporter: Jacob Ferriero
>Assignee: Jacob Ferriero
>Priority: Trivial
>
> Often times airflow operators (derived from BaseOperator) kick-off a long 
> running tasks and then waits / polls, blocking a worker slot until the long 
> running task completes. This can be problematic in environments with many 
> long running tasks.
> BaseSensorOperator was improved by implementing `reschedule` mode to solve 
> the similar issue with long running sensors blocking a worker to poll for a 
> long time.
> This issue is to track how we could provide a primitive that would make it 
> easy to develop operators for long running tasks that reschedule a `poll` 
> operation rather than blocking in their `execute` method.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj edited a comment on issue #6210: [AIRFLOW-5567] prototype BaseAsyncOperator

[GitBox]

mik-laj edited a comment on issue #6210: [AIRFLOW-5567] prototype 
BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#issuecomment-536221991
 
 
   My team was thinking of creating similar features but we haven't started 
work and it's possible that the plans will change. However, I will gladly help 
in implementing this features in the community.
   
   Some observation that I can add now it is useful that the functionality of 
the code execution after the operation is completed.
   For example:
   * Pre-execute: create a report request in Google Marketing Platform
   * Poke: Waiting for status change from processing to completed
   * Post-execute: download the raport to GCS
   
   The current implementation lacks a mechanism for passing the context of 
operations to the next step, e.g. the ID of the created object.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #6210: [AIRFLOW-5567] prototype BaseAsyncOperator

[GitBox]

mik-laj commented on issue #6210: [AIRFLOW-5567] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#issuecomment-536221991
 
 
   My team was thinking of creating similar features.  Some observation that I 
can add now it is useful that the functionality of the code execution after the 
operation is completed.
   For example:
   * Pre-execute: create a report request in Google Marketing Platform
   * Poke: Waiting for status change from processing to completed
   * Post-execute: download the raport to GCS
   
   The current implementation lacks a mechanism for passing the context of 
operations to the next step, e.g. the ID of the created object.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6204: [AIRFLOW-XXX] Use new import path in GCP table

[GitBox]

codecov-io commented on issue #6204: [AIRFLOW-XXX] Use new import path in GCP 
table
URL: https://github.com/apache/airflow/pull/6204#issuecomment-536221425
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=h1) 
Report
   > Merging 
[#6204](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6204/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6204  +/-   ##
   ==
   - Coverage   80.02%   80.01%   -0.02% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815028145   -5 
   - Misses   7026 7031   +5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6204/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `89.9% <0%> (-1.53%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=footer). 
Last update 
[2350b2f...b3dcde5](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6204: [AIRFLOW-XXX] Use new import path in GCP table

[GitBox]

codecov-io edited a comment on issue #6204: [AIRFLOW-XXX] Use new import path 
in GCP table
URL: https://github.com/apache/airflow/pull/6204#issuecomment-536221425
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=h1) 
Report
   > Merging 
[#6204](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6204/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6204  +/-   ##
   ==
   - Coverage   80.02%   80.01%   -0.02% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815028145   -5 
   - Misses   7026 7031   +5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6204/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `89.9% <0%> (-1.53%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=footer). 
Last update 
[2350b2f...b3dcde5](https://codecov.io/gh/apache/airflow/pull/6204?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jaketf opened a new pull request #6210: [AIRFLOW-5567] prototype BaseAsyncOperator

[GitBox]

jaketf opened a new pull request #6210: [AIRFLOW-5567] prototype 
BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-620) Ability to refresh logs of a TI without reloading whole page

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940151#comment-16940151
 ] 

Sergio Kef commented on AIRFLOW-620:


Please close duplicate when/if this gets resolved

> Ability to refresh logs of a TI without reloading whole page
> 
>
> Key: AIRFLOW-620
> URL: https://issues.apache.org/jira/browse/AIRFLOW-620
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Sumit Maheshwari
>Assignee: Sai Phanindhra
>Priority: Major
>  Labels: UI, usability
>
> As of now to see the latest logs of a task instance one has to refresh the 
> whole page, which is time consuming. It'll be great if just logs can be 
> re-fetched from backend. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj merged pull request #6193: [AIRFLOW-XXX] Add protocol operators and hooks table

[GitBox]

mik-laj merged pull request #6193: [AIRFLOW-XXX] Add protocol operators and 
hooks table
URL: https://github.com/apache/airflow/pull/6193
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-497) Release plans & info

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940147#comment-16940147
 ] 

Sergio Kef commented on AIRFLOW-497:


I believe the latest plan and decisions is depicted here: 
[https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Release+Planning+and+Supported+Release+Lifetime]

> Release plans & info
> 
>
> Key: AIRFLOW-497
> URL: https://issues.apache.org/jira/browse/AIRFLOW-497
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: core, documentation
>Reporter: Alexander Kachkaev
>Priority: Minor
>  Labels: build, newbie, release
>
> I did a couple of experiments with airflow several months ago and returned to 
> explore it properly this week. After a few days of quite intensive reading 
> and hacking it still remains unclear to me what's going on with the project 
> ATM.
> The latest release is 1.7.1.3, which dates back to 2016-06-13 (three months 
> from now). The docs on pythonhosted sometimes refer to 1.8 and git blame 
> reveals that these mentionings have been there since at least April 2016. 
> JIRA's dashboard has references to versions 1.8 and 2.0, but those only 
> contain lists with issues - no deadline etc.
> I imagine that core developers have a clear picture about the situation and 
> it is probably possible to figure things out from the mailing list and 
> gitter, However, it would be good to see roadmap etc. in a slightly more 
> accessible way.
> More frequent releases will help a lot as well. I'm seeing some issues when 
> running 1.7.1.3 via docker-airflow / celery, but it's totally unclear whether 
> these still exist on airflow's master branch or even something's wrong with 
> the docker wrapper I'm using. Opening an issue in JIRA seems somewhat stupid 
> in this situation.
> Could anyone please increase the clarity of meta?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-487) Split/Chunk operator

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940145#comment-16940145
 ] 

Sergio Kef commented on AIRFLOW-487:


I don't understand how this would look like. Can one give an example?

> Split/Chunk operator
> 
>
> Key: AIRFLOW-487
> URL: https://issues.apache.org/jira/browse/AIRFLOW-487
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: mohsen
>Priority: Major
>
> it would be nice to have a split (python) operator, which takes a job that 
> returns a list and streams each item to the specified job. like this pattern: 
> http://www.workflowpatterns.com/patterns/control/basic/wcp2.php
> also, a chunk operator that splits a list to smaller chunks is more general 
> solution
> i don't know if i should post this feature request here, or if there is any 
> try on this already. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5172) Add ability to have DAGs execute at the start of their scheduled interval

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940142#comment-16940142
 ] 

Sergio Kef commented on AIRFLOW-5172:
-

Please close duplicates as well when (if) this one gets merged

> Add ability to have DAGs execute at the start of their scheduled interval
> -
>
> Key: AIRFLOW-5172
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5172
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: DAG, scheduler
>Affects Versions: 1.10.4
>Reporter: Ian Roddis
>Priority: Minor
>
> Airflow's scheduling of tasks at the end of their interval can be confusing, 
> and difficult to choreograph in cases where tasks must execute at a specific 
> time (e.g. collecting ephemeral data).
> I'm sure this feature has been requested before, but my Jira search skills 
> aren't good enough to find much on it.
> This issue to add an option to schedule DAGs at the beginning of the 
> scheduled interval.
> I'll be submitting a PR to add this feature will be arriving shortly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-433) Allow tasks to be scheduled in the current schedule interval instead of previous

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940141#comment-16940141
 ] 

Sergio Kef commented on AIRFLOW-433:


Seems that 5172 is being worked on resolving this

> Allow tasks to be scheduled in the current schedule interval instead of 
> previous
> 
>
> Key: AIRFLOW-433
> URL: https://issues.apache.org/jira/browse/AIRFLOW-433
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: scheduler
>Reporter: Vineet Goel
>Priority: Major
>
> I understand that airflow chose to schedule dag runs for the previous 
> schedule interval because of the way data pipelines for batch processing are 
> usually designed. For example, currently a daily job is scheduled for a dag 
> run with execution date of Monday on a Tuesday. 
> However this is a bit weird for jobs that need to provide the same day of 
> running as the execution_date (or {{ ds }}). I can get around this by using 
> the date_add macro function for daily jobs but this workaround fails in the 
> case of jobs that are scheduled to run on weekdays and should provide the 
> same day in the context.
> It might be a good idea to make this configurable for a given dag as to 
> whether we want the scheduler to schedule for the same interval or the 
> previous. I understand by reading the code that this might be a bigger 
> feature addition than it seems as this might involve major changes to how the 
> scheduler works but this should be a good feature to have as airflow is a 
> great tool to use for non batch processing jobs as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-271) schedule_interval at a particular time behaves strangely

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940140#comment-16940140
 ] 

Sergio Kef commented on AIRFLOW-271:


This comes a bit late, but I think this is how the scheduler works:

The execution happens when last_execution (or start_date) + interval passes.

So

you start on 20/06, interval is daily, period window closes on 21/06,  "now" is 
23/06 so it gets triggered

next instance is 21/06, interval is daily, period window closes on 22/06, "now" 
is 23/06 so it gets triggered

next instance is 22/06, interval is daily, period window closes on 23/06 13:01, 
"now" is 23/06 07:00 so it won't get triggered till window closes.

For more details 
https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls

> schedule_interval at a particular time behaves strangely
> 
>
> Key: AIRFLOW-271
> URL: https://issues.apache.org/jira/browse/AIRFLOW-271
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.7.1
>Reporter: TERESA YAN
>Priority: Major
> Fix For: 1.7.1.2
>
> Attachments: feed_scheduler_template.py
>
>
> I have created a dag with the following configs in a python dag script.
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2016,6,20),
> 'email': email_list,
> 'email_on_failure': True,
> 'email_on_retry': True,
> 'retries': 3,
> 'retry_delay': timedelta(minutes=2),
> 'provide_context': True
> }
> dag = DAG('feed_scheduler_template', default_args=default_args, 
> schedule_interval="01 16 * * *")
> When I run the scheduler,  it gives a strange behavior, for example today is 
> 6/20 19:30  (I clear the db when I run the scheduler), start_date is 6/20
> It will start running for the following three timestamps in the logs directory
> data@dp-i-54a2648f:~/airflow/logs/feed_scheduler_template $ ls -l send
> total 12
> -rw-rw-r-- 1 data data 3099 Jun 22 19:30 2016-06-20T00:00:00
> -rw-rw-r-- 1 data data 3100 Jun 22 19:30 2016-06-20T16:01:00
> -rw-rw-r-- 1 data data 3100 Jun 22 19:30 2016-06-21T16:01:00
> The question is
> 1.  Why is 2016-06-20T00:00:00 at 0 hour 0 minute get executed because I only 
> want 16:01.
> 2.  I never get the 2016-06-22T16:01:00 run although my machine time already 
> pass that 16:01 hour on June 22.
> Any idea?
> Thanks so much



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-3553) Microseconds in manually triggered tasks break "mark as success"

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940137#comment-16940137
 ] 

Sergio Kef commented on AIRFLOW-3553:
-

This is a known issue due to the difference of timestamps between airflow and 
various rdbms

> Microseconds in manually triggered tasks break "mark as success" 
> -
>
> Key: AIRFLOW-3553
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3553
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.9.0, 1.10.1, 2.0.0
>Reporter: Jozef Fekiac
>Priority: Major
>
>  
> If a user wants to mark success on a dagRun with microseconds (I.e. manually 
> triggered from GUI), user can't mark tasks as success. 
>  
> in 1.9, replace microseconds is a default behaviour, as per code, but 
> [https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/www/views.py#L915]
> doesn't handle the microseconds, setting run_id
> propagating it to
> [https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/api/common/experimental/trigger_dag.py#L37]
> resulting in no run_id alteration, => it's with microseconds, disabling the 
> "mark as success function" from GUI



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #6202: [AIRFLOW-XXX] Improve ASF operators table

[GitBox]

codecov-io edited a comment on issue #6202: [AIRFLOW-XXX] Improve ASF operators 
table
URL: https://github.com/apache/airflow/pull/6202#issuecomment-536217904
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=h1) 
Report
   > Merging 
[#6202](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **decrease** coverage by `0.4%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6202/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6202  +/-   ##
   ==
   - Coverage   80.02%   79.62%   -0.41% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815028009 -141 
   - Misses   7026 7167 +141
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==)
 | `47.61% <0%> (-52.39%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `77.96% <0%> (-15.26%)` | :arrow_down: |
   | 
[airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=)
 | `81.81% <0%> (-11.37%)` | :arrow_down: |
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `85% <0%> (-5%)` | :arrow_down: |
   | 
[airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=)
 | `63.26% <0%> (-4.09%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `53.79% <0%> (-3.11%)` | :arrow_down: |
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `75.82% <0%> (-1.79%)` | :arrow_down: |
   | 
[airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=)
 | `73.19% <0%> (-1.21%)` | :arrow_down: |
   | ... and [3 
more](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=footer). 
Last update 
[2350b2f...740b8b7](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6202: [AIRFLOW-XXX] Improve ASF operators table

[GitBox]

codecov-io commented on issue #6202: [AIRFLOW-XXX] Improve ASF operators table
URL: https://github.com/apache/airflow/pull/6202#issuecomment-536217904
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=h1) 
Report
   > Merging 
[#6202](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **decrease** coverage by `0.4%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6202/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6202  +/-   ##
   ==
   - Coverage   80.02%   79.62%   -0.41% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815028009 -141 
   - Misses   7026 7167 +141
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==)
 | `47.61% <0%> (-52.39%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `77.96% <0%> (-15.26%)` | :arrow_down: |
   | 
[airflow/utils/log/colored\_log.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9sb2cvY29sb3JlZF9sb2cucHk=)
 | `81.81% <0%> (-11.37%)` | :arrow_down: |
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `85% <0%> (-5%)` | :arrow_down: |
   | 
[airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=)
 | `63.26% <0%> (-4.09%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `53.79% <0%> (-3.11%)` | :arrow_down: |
   | 
[airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5)
 | `75.82% <0%> (-1.79%)` | :arrow_down: |
   | 
[airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=)
 | `73.19% <0%> (-1.21%)` | :arrow_down: |
   | ... and [3 
more](https://codecov.io/gh/apache/airflow/pull/6202/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=footer). 
Last update 
[2350b2f...740b8b7](https://codecov.io/gh/apache/airflow/pull/6202?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-391) Airflow on Pypi still links to the AirBnB repo

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940134#comment-16940134
 ] 

Sergio Kef commented on AIRFLOW-391:


Now links to [https://github.com/apache/incubator-airflow] instead of 
[https://github.com/apache/airflow|https://github.com/apache/incubator-airflow] 
:D but the redirect is in place.

Let's close this one.

> Airflow on Pypi still links to the AirBnB repo
> --
>
> Key: AIRFLOW-391
> URL: https://issues.apache.org/jira/browse/AIRFLOW-391
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Sean Cronin
>Priority: Trivial
>
> https://pypi.python.org/pypi/airflow/ still links to 
> https://github.com/airbnb/airflow instead of 
> https://github.com/apache/incubator-airflow



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-384) Use params property for all request types in HTTP Hook

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940133#comment-16940133
 ] 

Sergio Kef commented on AIRFLOW-384:


This is already implemented. Can we close this?

> Use params property for all request types in HTTP Hook
> --
>
> Key: AIRFLOW-384
> URL: https://issues.apache.org/jira/browse/AIRFLOW-384
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Reporter: Christopher Quinones
>Assignee: Christopher Quinones
>Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> In order to pass params to http hooks, one is required to add them directly 
> to the endpoint.  The requests package already supports passing a dictionary 
> of params into it to avoid adding extra logic at the DAG level.
> Adopt params in http_hook.py.  This also ensures that params are not printed 
> to the console.  In some of my use cases, parameters are secure and should 
> not be printed into the airflow DAG output.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (AIRFLOW-380) Cannot cancel "catch-up" dag runs after unpausing a dag (or restarting scheduler after long)

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940132#comment-16940132
 ] 

Sergio Kef edited comment on AIRFLOW-380 at 9/28/19 6:41 PM:
-

I think Airflow logic is to catch up (because it follows the very simple rule 
while start date + interval <= now, create new task instance).

As mentioned above, one can mark all as succeeded, or even rename the dag to v2 
with a new start_date.

[~vineetgoel] do you wish to keep this open?


was (Author: serkef):
I think Airflow logic is to catch up (because it follows the very simple rule 
while start date + interval <= now, create new task instance).

As mentioned above, one can mark all as succeeded, or even rename the dag to v2 
with a new start_date.

> Cannot cancel "catch-up" dag runs after unpausing a dag (or restarting 
> scheduler after long)
> 
>
> Key: AIRFLOW-380
> URL: https://issues.apache.org/jira/browse/AIRFLOW-380
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Vineet Goel
>Priority: Major
>
> Currently when we unpause a dag or bring the scheduler back up after long, 
> the scheduler starts scheduler dag runs for each execution in between (for 
> the time that has elapsed). There should be a way to cancel these "backfills" 
> if they are not really required.
> While this is simpler to clear using the cli if the dagruns are created 
> simultaneously, it becomes a bit hard if the max_active_runs for the dag is 
> 1. This way, we need to clear each dag run individually which isn't ideal. 
> One workaround is to keep start_date as datetime.now() and deleting past dag 
> runs from the database prior to unpausing the dag. However this isn't ideal.
> Is there a way of doing that I don't know about?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io commented on issue #6201: [AIRFLOW-XXX] Improve style - missing commas, redundant dots

[GitBox]

codecov-io commented on issue #6201: [AIRFLOW-XXX] Improve style - missing 
commas, redundant dots
URL: https://github.com/apache/airflow/pull/6201#issuecomment-536216154
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6201?src=pr&el=h1) 
Report
   > Merging 
[#6201](https://codecov.io/gh/apache/airflow/pull/6201?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6201/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6201?src=pr&el=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#6201   +/-   ##
   ===
 Coverage   80.02%   80.02%   
   ===
 Files 610  610   
 Lines   3517635176   
   ===
 Hits2815028150   
 Misses   7026 7026
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6201?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6201?src=pr&el=footer). 
Last update 
[2350b2f...cf58720](https://codecov.io/gh/apache/airflow/pull/6201?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-372) DAGs can run before start_date time

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940130#comment-16940130
 ] 

Sergio Kef commented on AIRFLOW-372:


As stated in wiki, airflow is not designed to handle changes of start_date. 
[https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls]

Let's close this, unless one wants to come with an enhancement proposal.

> DAGs can run before start_date time
> ---
>
> Key: AIRFLOW-372
> URL: https://issues.apache.org/jira/browse/AIRFLOW-372
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.7.1.2
>Reporter: Isaac Steele
>Priority: Major
>
> If you turn off a DAG in the UI, there seemingly is no way to prevent 
> "missed" runs to schedule after the DAG is turned back on. I thought the 
> workaround for this, since it is not a parameterized option to prevent, would 
> be to update the start_date in the DAG code before turning the DAG back on. 
> This does not work, and therefore the scheduler is running dag_runs *before* 
> the listed start_date.
> To reproduce:
> # Create a DAG with a schedule_interval
> # Let the DAG run at least once
> # Turn off the DAG in the UI
> # Allow the schedule_interval to pass at least twice
> # Update the start_date in the DAG to be be after the two interval time
> # (I then removed the compiled python file and restarted airflow/scheduler 
> just to make sure)
> # Turn DAG back on in UI
> Result: All dag_runs that were "missed" while the DAG was turned off run, 
> despite the start_date being later.
> Ideally the start_date would always be honored. And also there would be a 
> parameter to just not run any "missed" dag_runs.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-380) Cannot cancel "catch-up" dag runs after unpausing a dag (or restarting scheduler after long)

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940132#comment-16940132
 ] 

Sergio Kef commented on AIRFLOW-380:


I think Airflow logic is to catch up (because it follows the very simple rule 
while start date + interval <= now, create new task instance).

As mentioned above, one can mark all as succeeded, or even rename the dag to v2 
with a new start_date.

> Cannot cancel "catch-up" dag runs after unpausing a dag (or restarting 
> scheduler after long)
> 
>
> Key: AIRFLOW-380
> URL: https://issues.apache.org/jira/browse/AIRFLOW-380
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Vineet Goel
>Priority: Major
>
> Currently when we unpause a dag or bring the scheduler back up after long, 
> the scheduler starts scheduler dag runs for each execution in between (for 
> the time that has elapsed). There should be a way to cancel these "backfills" 
> if they are not really required.
> While this is simpler to clear using the cli if the dagruns are created 
> simultaneously, it becomes a bit hard if the max_active_runs for the dag is 
> 1. This way, we need to clear each dag run individually which isn't ideal. 
> One workaround is to keep start_date as datetime.now() and deleting past dag 
> runs from the database prior to unpausing the dag. However this isn't ideal.
> Is there a way of doing that I don't know about?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj opened a new pull request #6209: [AIRFLOW-XXX][depends on #6193] Add service transfer operators

[GitBox]

mik-laj opened a new pull request #6209: [AIRFLOW-XXX][depends on #6193] Add 
service transfer operators
URL: https://github.com/apache/airflow/pull/6209
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (AIRFLOW-287) Implement SFTP hook

[Ash Berlin-Taylor (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor closed AIRFLOW-287.
-
Resolution: Done

Has existed for a while now.

> Implement SFTP hook
> ---
>
> Key: AIRFLOW-287
> URL: https://issues.apache.org/jira/browse/AIRFLOW-287
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
>Priority: Minor
>
> I would be useful to have SFTP hook similar to FTP hook we already have. The 
> *pysftp* library can be used underneath.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (AIRFLOW-299) Evaluate a plugin architecture to replace contrib

[Ash Berlin-Taylor (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor closed AIRFLOW-299.
-
Resolution: Done

We've both got plugins and are renaming everything in contrib.

> Evaluate a plugin architecture to replace contrib
> -
>
> Key: AIRFLOW-299
> URL: https://issues.apache.org/jira/browse/AIRFLOW-299
> Project: Apache Airflow
>  Issue Type: Task
>  Components: contrib
>Reporter: Dan Davydov
>Priority: Minor
>  Labels: contrib
>
> We should take a look at the usefulness of a plugin architecture similar to 
> tools like Jenkins for the contrib folder. This way we can delegate/scale 
> committers for contrib, keep the git history/repo size/tests of the repo 
> smaller, allow users to publish operators without waiting for their commits 
> to be merged by a committer etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj opened a new pull request #6208: [AIRFLOW-XXX][depends on #6193] Add protocol transfer operators

[GitBox]

mik-laj opened a new pull request #6208: [AIRFLOW-XXX][depends on #6193] Add 
protocol transfer operators
URL: https://github.com/apache/airflow/pull/6208
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-299) Evaluate a plugin architecture to replace contrib

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940126#comment-16940126
 ] 

Sergio Kef commented on AIRFLOW-299:


I would agree to remove "extras" from airflow and keep the core scheduler. That 
said, there is already AIP-21 
[https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths]
 which decides on merging the contrib with the main modules.

I think this ticket can close, and a new AIP can be created if one feels like 
proposing a new restructuring

> Evaluate a plugin architecture to replace contrib
> -
>
> Key: AIRFLOW-299
> URL: https://issues.apache.org/jira/browse/AIRFLOW-299
> Project: Apache Airflow
>  Issue Type: Task
>  Components: contrib
>Reporter: Dan Davydov
>Priority: Minor
>  Labels: contrib
>
> We should take a look at the usefulness of a plugin architecture similar to 
> tools like Jenkins for the contrib folder. This way we can delegate/scale 
> committers for contrib, keep the git history/repo size/tests of the repo 
> smaller, allow users to publish operators without waiting for their commits 
> to be merged by a committer etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5342) Can't run initdb using Microsoft SQL server

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940125#comment-16940125
 ] 

ASF subversion and git services commented on AIRFLOW-5342:
--

Commit 0bbdc7a888f08efdf32775ae29171f8ec49f8a57 in airflow's branch 
refs/heads/v1-10-test from Morten Post
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0bbdc7a ]

[AIRFLOW-5342] Fix MSSQL breaking task_instance db migration

MSSQL does not allow altering columns to NOT NULL when the column
is used in an index. Therefore we drop the ti_pool index and recreate
it after modifying the column.

Co-authored-by: mattinbits <3765307+mattinb...@users.noreply.github.com>
Co-authored-by: sirVir 
(cherry picked from commit f1674e30d36c617e95e3b3762739197932726f4e)


> Can't run initdb using Microsoft SQL server
> ---
>
> Key: AIRFLOW-5342
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5342
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration, core, database
>Affects Versions: 1.10.4
>Reporter: Adam Trump
>Assignee: Morten Post
>Priority: Critical
> Fix For: 1.10.6
>
>
> I'm using Microsoft SQL server 2016 as metadata db (for the 
> sql_alchemy_conn), and pymssql as the Python driver.
> When running airflow initdb, airflow creates a table named `TaskInstance` 
> with column `pool` that is initially Nullable (when the table is created).
> Later, an index named `ti_pool` on a few columns, including `pool` column in 
> table `TaskInstance`.
> Then, airflow will try to alter table `TaskInstance` and change the column 
> `pool` to `NOT NULL`.
> This does not work on Microsoft SQL Server since a column with an index 
> defined on it cannot be changed, unless the index is deleted before.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-287) Implement SFTP hook

[Sergio Kef (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16940124#comment-16940124
 ] 

Sergio Kef commented on AIRFLOW-287:


Can we close this?

> Implement SFTP hook
> ---
>
> Key: AIRFLOW-287
> URL: https://issues.apache.org/jira/browse/AIRFLOW-287
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
>Priority: Minor
>
> I would be useful to have SFTP hook similar to FTP hook we already have. The 
> *pysftp* library can be used underneath.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj opened a new pull request #6207: [AIRFLOW-XXX] Add software transfer operators

[GitBox]

mik-laj opened a new pull request #6207: [AIRFLOW-XXX] Add software transfer 
operators
URL: https://github.com/apache/airflow/pull/6207
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new pull request #6206: [AIRFLOW-XXX] Add more GCP transfer operators

[GitBox]

mik-laj opened a new pull request #6206: [AIRFLOW-XXX] Add more GCP transfer 
operators
URL: https://github.com/apache/airflow/pull/6206
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new pull request #6205: [AIRFLOW-XXX] Add more AWS transfer operators

[GitBox]

mik-laj opened a new pull request #6205: [AIRFLOW-XXX] Add more AWS transfer 
operators
URL: https://github.com/apache/airflow/pull/6205
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new pull request #6204: [AIRFLOW-XXX] Use new import path in GCP table

[GitBox]

mik-laj opened a new pull request #6204: [AIRFLOW-XXX] Use new import path in 
GCP table
URL: https://github.com/apache/airflow/pull/6204
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new pull request #6203: [AIRFLOW-XXX] Add more ASF transfer operators

[GitBox]

mik-laj opened a new pull request #6203: [AIRFLOW-XXX] Add more ASF transfer 
operators
URL: https://github.com/apache/airflow/pull/6203
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new pull request #6202: [AIRFLOW-XXX] Improve ASF operators table

[GitBox]

mik-laj opened a new pull request #6202: [AIRFLOW-XXX] Improve ASF operators 
table
URL: https://github.com/apache/airflow/pull/6202
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new pull request #6201: [AIRFLOW-XXX] Improve style - missing comma, redundant dot

[GitBox]

mik-laj opened a new pull request #6201: [AIRFLOW-XXX] Improve style - missing 
comma, redundant dot
URL: https://github.com/apache/airflow/pull/6201
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6193: [AIRFLOW-XXX] Add protocol operators and hooks table

[GitBox]

codecov-io edited a comment on issue #6193: [AIRFLOW-XXX] Add protocol 
operators and hooks table
URL: https://github.com/apache/airflow/pull/6193#issuecomment-536152254
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6193?src=pr&el=h1) 
Report
   > Merging 
[#6193](https://codecov.io/gh/apache/airflow/pull/6193?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2350b2fcdca90004cc724c577fd262cac9028ebf?src=pr&el=desc)
 will **decrease** coverage by `0.96%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6193/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/6193?src=pr&el=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6193  +/-   ##
   ==
   - Coverage   80.02%   79.05%   -0.97% 
   ==
 Files 610  610  
 Lines   3517635176  
   ==
   - Hits2815027809 -341 
   - Misses   7026 7367 +341
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6193?src=pr&el=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6193/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6193/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6193/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6193/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6193/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6193/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6193/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6193/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6193/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `40.74% <0%> (-35.56%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6193/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | ... and [18 
more](https://codecov.io/gh/apache/airflow/pull/6193/diff?src=pr&el=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6193?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6193?src=pr&el=footer). 
Last update 
[2350b2f...a5cec6e](https://codecov.io/gh/apache/airflow/pull/6193?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj edited a comment on issue #6193: [AIRFLOW-XXX] Add protocol operators and hooks table

[GitBox]

mik-laj edited a comment on issue #6193: [AIRFLOW-XXX] Add protocol operators 
and hooks table
URL: https://github.com/apache/airflow/pull/6193#issuecomment-536201717
 
 
   Rebased. Now I can start working on transfer operators for protocol, 
services and services.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #6193: [AIRFLOW-XXX][depends on #6191] Add protocol operators and hooks table

[GitBox]

mik-laj commented on issue #6193: [AIRFLOW-XXX][depends on #6191] Add protocol 
operators and hooks table
URL: https://github.com/apache/airflow/pull/6193#issuecomment-536201717
 
 
   Rebased


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle merged pull request #6191: [AIRFLOW-XXX] Add software integration tables

[GitBox]

feluelle merged pull request #6191: [AIRFLOW-XXX] Add software integration 
tables
URL: https://github.com/apache/airflow/pull/6191
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #6191: [AIRFLOW-XXX] Add software integration tables

[GitBox]

mik-laj commented on a change in pull request #6191: [AIRFLOW-XXX] Add software 
integration tables
URL: https://github.com/apache/airflow/pull/6191#discussion_r329309707
 
 

 ##
 File path: docs/integration.rst
 ##
 @@ -866,3 +866,124 @@ These integrations allow you to perform various 
operations within various servic
  - :mod:`airflow.hooks.zendesk_hook`
  -
  -
+
+.. _software:
+
+Software integrations
+-
+
+Operators and Hooks
+'''
+
+Software operators and hooks
+
+
+These integrations allow you to perform various operations using various 
software.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Service name
+ - Guide
+ - Hook
+ - Operators
+ - Sensors
+
+   * - `Celery `__
+ -
+ -
+ -
+ - :mod:`airflow.contrib.sensors.celery_queue_sensor`
+
+   * - `Docker `__
+ -
+ - :mod:`airflow.hooks.docker_hook`
+ - :mod:`airflow.operators.docker_operator`,
+   :mod:`airflow.contrib.operators.docker_swarm_operator`
+ -
+
+   * - `GNU Bash `__
+ - :doc:`How to use `
+ -
+ - :mod:`airflow.operators.bash_operator`
+ - :mod:`airflow.contrib.sensors.bash_sensor`
+
+   * - `Kubernetes `__
+ - :doc:`How to use `
+ -
+ - :mod:`airflow.contrib.operators.kubernetes_pod_operator`
+ -
+
+   * - `Microsoft SQL Server (MSSQL) 
`__
+ -
+ - :mod:`airflow.hooks.mssql_hook`
+ - :mod:`airflow.operators.mssql_operator`
+ -
+
+   * - `MongoDB `__
+ -
+ - :mod:`airflow.contrib.hooks.mongo_hook`
+ -
+ - :mod:`airflow.contrib.sensors.mongo_sensor`
+
+
+   * - `MySQL `__
+ -
+ - :mod:`airflow.hooks.mysql_hook`
+ - :mod:`airflow.operators.mysql_operator`
+ -
+
+   * - `OpenFaaS `__
+ -
+ - :mod:`airflow.contrib.hooks.openfaas_hook`
+ -
+ -
+
+   * - `Oracle `__
+ -
+ - :mod:`airflow.hooks.oracle_hook`
+ - :mod:`airflow.operators.oracle_operator`
+ -
+
+   * - `Papermill `__
+ - :doc:`How to use `
+ -
+ - :mod:`airflow.operators.papermill_operator`
+ -
+
+   * - `PostgresSQL `__
+ -
+ - :mod:`airflow.hooks.postgres_hook`
+ - :mod:`airflow.operators.postgres_operator`
+ -
+
+   * - `Presto `__
+ -
+ - :mod:`airflow.hooks.presto_hook`
+ - :mod:`airflow.operators.presto_check_operator`
+ -
+
+   * - `Python `__
+ -
+ -
+ - :mod:`airflow.operators.python_operator`
+ - :mod:`airflow.contrib.sensors.python_sensor`
+
+   * - `Redis `__
+ -
+ - :mod:`airflow.contrib.hooks.redis_hook`
+ - :mod:`airflow.contrib.operators.redis_publish_operator`
+ - :mod:`airflow.contrib.sensors.redis_pub_sub_sensor`,
+   :mod:`airflow.contrib.sensors.redis_key_sensor`.
 
 Review comment:
   Updated. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5553) mysql_to_gcs.MySqlToGoogleCloudStorageOperator binary values cannot be encoded whenever a BQ schema is not explicitly specified

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939965#comment-16939965
 ] 

ASF subversion and git services commented on AIRFLOW-5553:
--

Commit 22ef5ddbbbfe119050bedb986ebfd3f1d2c9b6ae in airflow's branch 
refs/heads/master from Nikolay
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=22ef5dd ]

[AIRFLOW-5553] Update mysql_to_gcs `bytes` check (#6183)

The check for `bytes` type is now based on value type too, not just BQ schema.

> mysql_to_gcs.MySqlToGoogleCloudStorageOperator binary values cannot be 
> encoded whenever a BQ schema is not explicitly specified
> ---
>
> Key: AIRFLOW-5553
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5553
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.3, 1.10.4, 1.10.5
>Reporter: Nikolay Stoynov
>Assignee: Nikolay Stoynov
>Priority: Minor
>  Labels: easyfix
>
>  
> The MySqlGoogleCloudStorageOperator currently deals with `bytes` values. 
> However, the way it detects when a value is of type `bytes` is dependent on 
> the column type in the BQ schema provided.
> Therefore, whenever a BQ schema is not provided, the value is not dealt with 
> and a serialization issue occurs:
> {noformat}
> [2018-10-11 17:59:36,056] INFO 
> airflow.task.task_runner.bash_task_runner.BashTaskRunner _read_task_logs - 
> Job 11906: Subtask my_subtask TypeError: Object of type 'bytes' is not JSON 
> serializable{noformat}
> This should be a trivial fix but needs to be addressed so that the check for 
> a bytes value is based on the type of the variable too.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] feluelle commented on issue #6200: Allow user to specify webserver_config.py path via env or configurati…

[GitBox]

feluelle commented on issue #6200: Allow user to specify webserver_config.py 
path via env or configurati…
URL: https://github.com/apache/airflow/pull/6200#issuecomment-536179297
 
 
   Welcome @eugene-chernyshenko !
   
   Please create a Jira Ticket addressing your changes. Thanks :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5553) mysql_to_gcs.MySqlToGoogleCloudStorageOperator binary values cannot be encoded whenever a BQ schema is not explicitly specified

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939964#comment-16939964
 ] 

ASF GitHub Bot commented on AIRFLOW-5553:
-

feluelle commented on pull request #6183: [AIRFLOW-5553] Bytes value check done 
based on the type of the variable
URL: https://github.com/apache/airflow/pull/6183
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> mysql_to_gcs.MySqlToGoogleCloudStorageOperator binary values cannot be 
> encoded whenever a BQ schema is not explicitly specified
> ---
>
> Key: AIRFLOW-5553
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5553
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.3, 1.10.4, 1.10.5
>Reporter: Nikolay Stoynov
>Assignee: Nikolay Stoynov
>Priority: Minor
>  Labels: easyfix
>
>  
> The MySqlGoogleCloudStorageOperator currently deals with `bytes` values. 
> However, the way it detects when a value is of type `bytes` is dependent on 
> the column type in the BQ schema provided.
> Therefore, whenever a BQ schema is not provided, the value is not dealt with 
> and a serialization issue occurs:
> {noformat}
> [2018-10-11 17:59:36,056] INFO 
> airflow.task.task_runner.bash_task_runner.BashTaskRunner _read_task_logs - 
> Job 11906: Subtask my_subtask TypeError: Object of type 'bytes' is not JSON 
> serializable{noformat}
> This should be a trivial fix but needs to be addressed so that the check for 
> a bytes value is based on the type of the variable too.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] feluelle merged pull request #6183: [AIRFLOW-5553] Bytes value check done based on the type of the variable

[GitBox]

feluelle merged pull request #6183: [AIRFLOW-5553] Bytes value check done based 
on the type of the variable
URL: https://github.com/apache/airflow/pull/6183
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6191: [AIRFLOW-XXX] Add software integration tables

[GitBox]

feluelle commented on a change in pull request #6191: [AIRFLOW-XXX] Add 
software integration tables
URL: https://github.com/apache/airflow/pull/6191#discussion_r329306885
 
 

 ##
 File path: docs/integration.rst
 ##
 @@ -866,3 +866,124 @@ These integrations allow you to perform various 
operations within various servic
  - :mod:`airflow.hooks.zendesk_hook`
  -
  -
+
+.. _software:
+
+Software integrations
+-
+
+Operators and Hooks
+'''
+
+Software operators and hooks
+
+
+These integrations allow you to perform various operations using various 
software.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Service name
+ - Guide
+ - Hook
+ - Operators
+ - Sensors
+
+   * - `Celery `__
+ -
+ -
+ -
+ - :mod:`airflow.contrib.sensors.celery_queue_sensor`
+
+   * - `Docker `__
+ -
+ - :mod:`airflow.hooks.docker_hook`
+ - :mod:`airflow.operators.docker_operator`,
+   :mod:`airflow.contrib.operators.docker_swarm_operator`
+ -
+
+   * - `GNU Bash `__
+ - :doc:`How to use `
+ -
+ - :mod:`airflow.operators.bash_operator`
+ - :mod:`airflow.contrib.sensors.bash_sensor`
+
+   * - `Kubernetes `__
+ - :doc:`How to use `
+ -
+ - :mod:`airflow.contrib.operators.kubernetes_pod_operator`
+ -
+
+   * - `Microsoft SQL Server (MSSQL) 
`__
+ -
+ - :mod:`airflow.hooks.mssql_hook`
+ - :mod:`airflow.operators.mssql_operator`
+ -
+
+   * - `MongoDB `__
+ -
+ - :mod:`airflow.contrib.hooks.mongo_hook`
+ -
+ - :mod:`airflow.contrib.sensors.mongo_sensor`
+
+
+   * - `MySQL `__
+ -
+ - :mod:`airflow.hooks.mysql_hook`
+ - :mod:`airflow.operators.mysql_operator`
+ -
+
+   * - `OpenFaaS `__
+ -
+ - :mod:`airflow.contrib.hooks.openfaas_hook`
+ -
+ -
+
+   * - `Oracle `__
+ -
+ - :mod:`airflow.hooks.oracle_hook`
+ - :mod:`airflow.operators.oracle_operator`
+ -
+
+   * - `Papermill `__
+ - :doc:`How to use `
+ -
+ - :mod:`airflow.operators.papermill_operator`
+ -
+
+   * - `PostgresSQL `__
+ -
+ - :mod:`airflow.hooks.postgres_hook`
+ - :mod:`airflow.operators.postgres_operator`
+ -
+
+   * - `Presto `__
+ -
+ - :mod:`airflow.hooks.presto_hook`
+ - :mod:`airflow.operators.presto_check_operator`
+ -
+
+   * - `Python `__
+ -
+ -
+ - :mod:`airflow.operators.python_operator`
+ - :mod:`airflow.contrib.sensors.python_sensor`
+
+   * - `Redis `__
+ -
+ - :mod:`airflow.contrib.hooks.redis_hook`
+ - :mod:`airflow.contrib.operators.redis_publish_operator`
+ - :mod:`airflow.contrib.sensors.redis_pub_sub_sensor`,
+   :mod:`airflow.contrib.sensors.redis_key_sensor`.
 
 Review comment:
   ```suggestion
  :mod:`airflow.contrib.sensors.redis_key_sensor`
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4823) Add the ability to toggle creation of default connections on deployment

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939918#comment-16939918
 ] 

ASF GitHub Bot commented on AIRFLOW-4823:
-

stale[bot] commented on pull request #5443: [AIRFLOW-4823] modified the source 
to have create_default_connections flag in config
URL: https://github.com/apache/airflow/pull/5443
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add the ability to toggle creation of default connections on deployment
> ---
>
> Key: AIRFLOW-4823
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4823
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.10.3
>Reporter: Richard Jarvis
>Priority: Minor
>
> Desire the capability to allow user to decide if default connections are 
> created or not when Airflow is deployed. 
>  
> Suggest addition of new property in config file



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] stale[bot] closed pull request #5443: [AIRFLOW-4823] modified the source to have create_default_connections flag in config

[GitBox]

stale[bot] closed pull request #5443: [AIRFLOW-4823] modified the source to 
have create_default_connections flag in config
URL: https://github.com/apache/airflow/pull/5443
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] robinedwards commented on issue #6195: [AIRFLOW-5560] Allow no confirmation on reset dags

[GitBox]

robinedwards commented on issue #6195: [AIRFLOW-5560] Allow no confirmation on 
reset dags
URL: https://github.com/apache/airflow/pull/6195#issuecomment-536165753
 
 
   I only chose `--yes` because it was already being used:
   
   https://github.com/apache/airflow/blob/master/airflow/bin/cli.py#L1711
   
   But I would be happy to change it to whatever you guys can agree on as long 
as its consistent
   
   Note that the PR for master also updates clear to also use `--yes` 
https://github.com/apache/airflow/pull/6197.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services