[jira] [Commented] (AIRFLOW-5566) Invalid deprecation warning for airflow home specification within config file
[
https://issues.apache.org/jira/browse/AIRFLOW-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945240#comment-16945240
]
ASF GitHub Bot commented on AIRFLOW-5566:
-
friedmanCadre commented on pull request #6270: [AIRFLOW-5566] Fix false
positive deprecation warning
URL: https://github.com/apache/airflow/pull/6270
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR fixes the following issue:
[This deprecation
warning](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L551-L568)
is run regardless of whether an 'airflow_home' entry exists in the config file
when the 'AIRFLOW_HOME' environment variables are set.
Traceback:
[config entry
check](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L551)
->
[config.has_option(...)](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L296-L304)
-> [get(...)](
https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L214-L267)
in get:
```
option = self._get_env_var_option(section, key)
if option is not None:
return option
```
'get' returns True, then 'has_option' returns True, which triggers the
warning
This PR adds an additional `file_has_option` method to the
`AirflowConfigParser` that does not check the environment variables for the
specified option.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
The additional method in this PR simply provides a way to access Python's
`ConfigParser` `has_option` method, which was overwritten in the
`AirflowConfigParser` class.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Invalid deprecation warning for airflow home specification within config file
> -
>
> Key: AIRFLOW-5566
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5566
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.10.5
>Reporter: Ian Friedman
>Assignee: Ian Friedman
>Priority: Minor
>
> [This deprecation
> warning|https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L551-L568]
> is run regardless of whether an 'airflow_home' entry exists in the config
> file when the 'AIRFLOW_HOME' environment variables are set.
[GitHub] [airflow] friedmanCadre opened a new pull request #6270: [AIRFLOW-5566] Fix false positive deprecation warning
friedmanCadre opened a new pull request #6270: [AIRFLOW-5566] Fix false
positive deprecation warning
URL: https://github.com/apache/airflow/pull/6270
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
This PR fixes the following issue:
[This deprecation
warning](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L551-L568)
is run regardless of whether an 'airflow_home' entry exists in the config file
when the 'AIRFLOW_HOME' environment variables are set.
Traceback:
[config entry
check](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L551)
->
[config.has_option(...)](https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L296-L304)
-> [get(...)](
https://github.com/apache/airflow/blob/4d4fda75333b1e6ae4e99b407ab2b1edc0d139d8/airflow/configuration.py#L214-L267)
in get:
```
option = self._get_env_var_option(section, key)
if option is not None:
return option
```
'get' returns True, then 'has_option' returns True, which triggers the
warning
This PR adds an additional `file_has_option` method to the
`AirflowConfigParser` that does not check the environment variables for the
specified option.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
The additional method in this PR simply provides a way to access Python's
`ConfigParser` `has_option` method, which was overwritten in the
`AirflowConfigParser` class.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6269: Use poke_interval of 0 to make EMR tests run green [AIRFLOW-XXX]
codecov-io commented on issue #6269: Use poke_interval of 0 to make EMR tests run green [AIRFLOW-XXX] URL: https://github.com/apache/airflow/pull/6269#issuecomment-538703206 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6269?src=pr=h1) Report > Merging [#6269](https://codecov.io/gh/apache/airflow/pull/6269?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr=desc) will **decrease** coverage by `0.02%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6269?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6269 +/- ## == - Coverage 80.05% 80.03% -0.03% == Files 610 610 Lines 3526435264 == - Hits2823228223 -9 - Misses 7032 7041 +9 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6269?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6269/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6269/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.24% <0%> (-0.51%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6269/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.55% <0%> (-0.35%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6269?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6269?src=pr=footer). Last update [32fd2af...1a45456](https://codecov.io/gh/apache/airflow/pull/6269?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] yegeniy opened a new pull request #6269: Use poke_interval of 0 to make EMR tests run green [AIRFLOW-XXX]
yegeniy opened a new pull request #6269: Use poke_interval of 0 to make EMR
tests run green [AIRFLOW-XXX]
URL: https://github.com/apache/airflow/pull/6269
```
As part of introducing EmrRunJobFlows in a separate PR, I noticed that
the EMR tests were so slow they were yellow and that a poke_interval of
0 is a valid poke_interval for the tests to run at.
Practically speaking, though, this is only saving a dozen seconds in
the test suite at most.
```
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
See above
### Tests
- [x] My PR ~adds the following unit tests __OR__~ does not need testing for
this extremely good reason:
I'm modifying existing tests only.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator
jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r331763207
##
File path: airflow/models/base_async_operator.py
##
@@ -0,0 +1,133 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from abc import abstractmethod
+from typing import Dict, List, Union, Optional
+
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+from airflow.models import SkipMixin, TaskReschedule
+from airflow.utils.decorators import apply_defaults
+
+PLACEHOLDER_RESOURCE_ID = 'RESOURCE_ID_NOT_APPLICABLE'
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must override the following methods:
+:meth:`submit_request`: fire a request for a long running operation
+:meth:`poke`: a method to check if the long running operation is
+complete it should return True when a success criteria is met.
+Optionally, AsyncOperators can override:
+:meth: `process_result` to perform any operations after the success
+criteria is met in :meth: `poke`
+:meth: `poke` is executed at a time interval and succeed when a
+criteria is met and fail if and when they time out. They are effctively
+an opinionated way use combine an Operator and a Sensor in order to kick
+off a long running process and poll for completion without tying up a
+worker slot while waiting between pokes leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+"""
+ui_color = '#9933ff' # type: str
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+@abstractmethod
+def submit_request(self, context) -> Optional[Union[str, List, Dict]]:
+"""
+This method should kick off a long running operation.
+This method should return the ID for the long running operation if
+applicable.
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+
+:returns: a resource_id for the long running operation.
+:rtype: Optional[Union[String, List, Dict]]
+"""
+raise AirflowException('Async Operators must override the
`submit_request` method.')
+
+def process_result(self, context):
+"""
+This method can optionally be overriden to process the result of a
long running operation.
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+"""
+self.log.info('Using default process_result. Got result of %s. Done.',
+ self.get_external_resource_id(context))
+
+def execute(self, context):
+# On the first execute call submit_request and set the
+# external resource id.
+task_reschedules = TaskReschedule.find_for_task_instance(context['ti'])
+if not task_reschedules:
+resource_id = self.submit_request(context)
+if not resource_id:
+resource_id = PLACEHOLDER_RESOURCE_ID
+self.set_external_resource_id(context, resource_id)
+
+super().execute(context)
+
+# TODO: find out why code below here never called.
+resource_id = self.get_external_resource_id(context)
+if resource_id == PLACEHOLDER_RESOURCE_ID:
+self.log.info("Calling
[GitHub] [airflow] jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator
jaketf commented on a change in pull request #6210: [AIRFLOW-5567] [Do not
Merge] prototype BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r33176
##
File path: airflow/models/base_async_operator.py
##
@@ -0,0 +1,133 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Base Asynchronous Operator for kicking off a long running
+operations and polling for completion with reschedule mode.
+"""
+from abc import abstractmethod
+from typing import Dict, List, Union, Optional
+
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.exceptions import AirflowException
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+from airflow.models import SkipMixin, TaskReschedule
+from airflow.utils.decorators import apply_defaults
+
+PLACEHOLDER_RESOURCE_ID = 'RESOURCE_ID_NOT_APPLICABLE'
+
+class BaseAsyncOperator(BaseSensorOperator, SkipMixin):
+"""
+AsyncOperators are derived from this class and inherit these attributes.
+
+AsyncOperators must override the following methods:
+:meth:`submit_request`: fire a request for a long running operation
+:meth:`poke`: a method to check if the long running operation is
+complete it should return True when a success criteria is met.
+Optionally, AsyncOperators can override:
+:meth: `process_result` to perform any operations after the success
+criteria is met in :meth: `poke`
+:meth: `poke` is executed at a time interval and succeed when a
+criteria is met and fail if and when they time out. They are effctively
+an opinionated way use combine an Operator and a Sensor in order to kick
+off a long running process and poll for completion without tying up a
+worker slot while waiting between pokes leveraging reschedule mode.
+
+:param soft_fail: Set to true to mark the task as SKIPPED on failure
+:type soft_fail: bool
+:param poke_interval: Time in seconds that the job should wait in
+between each tries
+:type poke_interval: int
+:param timeout: Time, in seconds before the task times out and fails.
+:type timeout: int
+"""
+ui_color = '#9933ff' # type: str
+
+@apply_defaults
+def __init__(self,
+ *args,
+ **kwargs) -> None:
+super().__init__(mode='reschedule', *args, **kwargs)
+
+@abstractmethod
+def submit_request(self, context) -> Optional[Union[str, List, Dict]]:
+"""
+This method should kick off a long running operation.
+This method should return the ID for the long running operation if
+applicable.
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+
+:returns: a resource_id for the long running operation.
+:rtype: Optional[Union[String, List, Dict]]
+"""
+raise AirflowException('Async Operators must override the
`submit_request` method.')
+
+def process_result(self, context):
+"""
+This method can optionally be overriden to process the result of a
long running operation.
+Context is the same dictionary used as when rendering jinja templates.
+
+Refer to get_template_context for more context.
+"""
+self.log.info('Using default process_result. Got result of %s. Done.',
+ self.get_external_resource_id(context))
+
+def execute(self, context):
+# On the first execute call submit_request and set the
+# external resource id.
+task_reschedules = TaskReschedule.find_for_task_instance(context['ti'])
+if not task_reschedules:
+resource_id = self.submit_request(context)
+if not resource_id:
+resource_id = PLACEHOLDER_RESOURCE_ID
+self.set_external_resource_id(context, resource_id)
+
+super().execute(context)
+
+# TODO: find out why code below here never called.
+resource_id = self.get_external_resource_id(context)
+if resource_id == PLACEHOLDER_RESOURCE_ID:
+self.log.info("Calling
[GitHub] [airflow] codecov-io commented on issue #6262: [AIRFLOW-5599] Improve Python 3 support in MLEngine integration
codecov-io commented on issue #6262: [AIRFLOW-5599] Improve Python 3 support in MLEngine integration URL: https://github.com/apache/airflow/pull/6262#issuecomment-538684323 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6262?src=pr=h1) Report > Merging [#6262](https://codecov.io/gh/apache/airflow/pull/6262?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr=desc) will **not change** coverage. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/6262?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#6262 +/- ## === Coverage 80.05% 80.05% === Files 610 610 Lines 3526435264 === Hits2823228232 Misses 7032 7032 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6262?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/utils/mlengine\_prediction\_summary.py](https://codecov.io/gh/apache/airflow/pull/6262/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfcHJlZGljdGlvbl9zdW1tYXJ5LnB5) | `0% <0%> (ø)` | :arrow_up: | | [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6262/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <100%> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6262?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6262?src=pr=footer). Last update [32fd2af...9134958](https://codecov.io/gh/apache/airflow/pull/6262?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #6266: Production image support including refactoring of build scripts
potiuk edited a comment on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538684102 Also - we will be asking community for help with documenting the official/prod image usage when the PR is ready. I hope we can get community to contribute on how they are using current prod images. I hope @mik-laj we can get some feedback and help from you/the people involved in improving the documentation. Looking forward to it. Any input and fixups with doc improvements are welcome. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6266: Production image support including refactoring of build scripts
potiuk commented on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538684102 Also - we will be asking community for help with documenting the official/prod image when the PR is ready. I hope we can get community to contribute on how they are using current prod images. I hope @mik-laj we can get some feedback and help from you/the people involved in improving the documentation. Looking forward to it. Any input and fixups with doc improvements are welcome. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6266: Production image support including refactoring of build scripts
potiuk commented on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538683562 Sure. This is still WIP/Draft. I will add documentation when it is ready. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests
codecov-io edited a comment on issue #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests URL: https://github.com/apache/airflow/pull/6268#issuecomment-538680046 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=h1) Report > Merging [#6268](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr=desc) will **increase** coverage by `0.19%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6268 +/- ## == + Coverage 80.05% 80.25% +0.19% == Files 610 610 Lines 3526435260 -4 == + Hits2823228299 +67 + Misses 7032 6961 -71 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6268/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvbWxlbmdpbmUucHk=) | `78.63% <100%> (+57.97%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=footer). Last update [32fd2af...36eb8d0](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests
codecov-io commented on issue #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests URL: https://github.com/apache/airflow/pull/6268#issuecomment-538680046 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=h1) Report > Merging [#6268](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr=desc) will **increase** coverage by `0.16%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6268 +/- ## == + Coverage 80.05% 80.22% +0.16% == Files 610 610 Lines 3526435260 -4 == + Hits2823228288 +56 + Misses 7032 6972 -60 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6268/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvbWxlbmdpbmUucHk=) | `78.63% <100%> (+57.97%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6268/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6268/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `89.9% <0%> (-1.53%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6268/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.55% <0%> (-0.35%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=footer). Last update [32fd2af...36eb8d0](https://codecov.io/gh/apache/airflow/pull/6268?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3978) Add missing types in MySqlToGoogleCloudStorageOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945135#comment-16945135
]
jack commented on AIRFLOW-3978:
---
The question is what happens if your MySQL table has BINARY field and you use
MySqlToGoogleCloudStorageOperator and then load it to BigQuery
Will it be recognized as {{BYTES}} in BigQuery with auto detect?
> Add missing types in MySqlToGoogleCloudStorageOperator
> --
>
> Key: AIRFLOW-3978
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3978
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Affects Versions: 1.10.2
>Reporter: Roster
>Assignee: Roster
>Priority: Minor
> Labels: gcs
>
> There fields are missing and can not be mapped:
> TIME, BINARY , VARBINARY
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #6267: [AIRFLOW-5601] Use built-in pagination mechanism in MLEngine hook
codecov-io commented on issue #6267: [AIRFLOW-5601] Use built-in pagination mechanism in MLEngine hook URL: https://github.com/apache/airflow/pull/6267#issuecomment-538677593 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6267?src=pr=h1) Report > Merging [#6267](https://codecov.io/gh/apache/airflow/pull/6267?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/32fd2afcf097c273618af122a9da65cf05b2ea93?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/6267?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6267 +/- ## == + Coverage 80.05% 80.06% +<.01% == Files 610 610 Lines 3526435260 -4 == - Hits2823228231 -1 + Misses 7032 7029 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6267?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/6267/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvbWxlbmdpbmUucHk=) | `21.36% <0%> (+0.7%)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6267/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `56.72% <0%> (-0.18%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6267?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6267?src=pr=footer). Last update [32fd2af...2b24c6f](https://codecov.io/gh/apache/airflow/pull/6267?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] YingboWang commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor
YingboWang commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor URL: https://github.com/apache/airflow/pull/5499#issuecomment-538674356 @seelmann We are deploying the smart sensor inside our company right now. Testing and monitoring it for further refinement. I will update the code for this PR and AIP document with the overall structure and some statistics based on our testing for the past several weeks. @BasPH @feng-tao, Thanks for your feedbacks. It's my bad not update this PR on time. The smart sensor operator itself is only used to create a control dag for taking over all sensors in the airflow cluster when the cluster is enable to "use_smart_sensor" by configuration. The smart sensor task is running as a service keep refreshing all active sensor tasks from DB and poking them. Users doesn't need to change their original sensor tasks when writing DAGs. When use smart sensor, the sensor tasks state changing is different with regular sensor. The scheduler will persist sensor's poke context to DB and set the state to "Smart_pending" so that the task will not be queued and taking a separate process in any worker. Smart sensor task instance which is a long running process on the work will refresh DB to get all sensors that are in "Smart" state (smart_running or smart_pending) and poke them. If the poke return True it will set the corresponding task_instance state to "success" so that the scheduler will be able to kick off the downstream task_instance in the next parsing loop. There is only a cluster configuration change for using smart sensor. No need to write DAGs in a different way. I will update the PR and the description with latest change and also update the AIP to include all details for smart sensor design. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #6266: Production image support including refactoring of build scripts
mik-laj edited a comment on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538674172 Can you add documentation? I would be happy with a short article similar to [Quick start](https://airflow.readthedocs.io/en/latest/start.html). It's ok for me to copy this article and replace the commands in this article. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6266: Production image support including refactoring of build scripts
mik-laj commented on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538674172 Can you add documentation? I would be happy with a short article similar to [quickstart(https://airflow.readthedocs.io/en/latest/start.html)]. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #6266: Production image support including refactoring of build scripts
mik-laj edited a comment on issue #6266: Production image support including refactoring of build scripts URL: https://github.com/apache/airflow/pull/6266#issuecomment-538674172 Can you add documentation? I would be happy with a short article similar to [Quick start](https://airflow.readthedocs.io/en/latest/start.html). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5602) Use unittest.mock in MLEngine hook tests
[
https://issues.apache.org/jira/browse/AIRFLOW-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945125#comment-16945125
]
ASF GitHub Bot commented on AIRFLOW-5602:
-
mik-laj commented on pull request #6268: [AIRFLOW-5602][depends on
AIRFLOW-5602] Use unittest.mock in MLEngine hook tests
URL: https://github.com/apache/airflow/pull/6268
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Use unittest.mock in MLEngine hook tests
>
>
> Key: AIRFLOW-5602
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5602
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp
>Affects Versions: 1.10.5
>Reporter: Kamil Bregula
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5601) Use built-in pagination mechanism in MLEngine hook
[
https://issues.apache.org/jira/browse/AIRFLOW-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945124#comment-16945124
]
ASF GitHub Bot commented on AIRFLOW-5601:
-
mik-laj commented on pull request #6267: [AIRFLOW-5601] Use built-in pagination
mechanism in MLEngine hook
URL: https://github.com/apache/airflow/pull/6267
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5601
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Use built-in pagination mechanism in MLEngine hook
> --
>
> Key: AIRFLOW-5601
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5601
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp
>Affects Versions: 1.10.5
>Reporter: Kamil Bregula
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #6268: [AIRFLOW-5602][depends on AIRFLOW-5602] Use unittest.mock in MLEngine hook tests
mik-laj opened a new pull request #6268: [AIRFLOW-5602][depends on
AIRFLOW-5602] Use unittest.mock in MLEngine hook tests
URL: https://github.com/apache/airflow/pull/6268
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj opened a new pull request #6267: [AIRFLOW-5601] Use built-in pagination mechanism in MLEngine hook
mik-laj opened a new pull request #6267: [AIRFLOW-5601] Use built-in pagination
mechanism in MLEngine hook
URL: https://github.com/apache/airflow/pull/6267
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5601
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5602) Use unittest.mock in MLEngine hook tests
Kamil Bregula created AIRFLOW-5602: -- Summary: Use unittest.mock in MLEngine hook tests Key: AIRFLOW-5602 URL: https://issues.apache.org/jira/browse/AIRFLOW-5602 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.5 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5601) Use built-in pagination mechanism in MLEngine hook
Kamil Bregula created AIRFLOW-5601: -- Summary: Use built-in pagination mechanism in MLEngine hook Key: AIRFLOW-5601 URL: https://issues.apache.org/jira/browse/AIRFLOW-5601 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.5 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] dimberman commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S pod to be configured from JSON/YAML file
dimberman commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S pod to be configured from JSON/YAML file URL: https://github.com/apache/airflow/pull/6230#discussion_r331754671 ## File path: airflow/executors/kubernetes_executor.py ## @@ -45,199 +47,6 @@ MAX_LABEL_LEN = 63 -class KubeConfig: # pylint: disable=too-many-instance-attributes Review comment: @davlum Could you move the Kubeconfig back and just make that a seperate PR? I'll get that merged quickly once this one goes through. I think with front-facing changes it better to put refactors in separately. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on issue #6237: AIRFLOW-5581: Join KubernetesJobWatcher in terminate call and unblock queues from blocking forever
dimberman commented on issue #6237: AIRFLOW-5581: Join KubernetesJobWatcher in terminate call and unblock queues from blocking forever URL: https://github.com/apache/airflow/pull/6237#issuecomment-538670434 @kpathak13 LGTM. Can you rebase? I'll merge when ready :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor
feng-tao commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor URL: https://github.com/apache/airflow/pull/5499#issuecomment-538661311 +1 on @BasPH , it will be good to change the default sensor behavior. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] seelmann commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor
seelmann commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor URL: https://github.com/apache/airflow/pull/5499#issuecomment-538650847 @YingboWang may I ask for the status of the smart sensor change? We are also running into issues with sensors being too heavy weight. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk opened a new pull request #6266: Production image support including refactoring of build scripts
potiuk opened a new pull request #6266: Production image support including
refactoring of build scripts
URL: https://github.com/apache/airflow/pull/6266
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
