[GitHub] [airflow] turbaszek commented on a change in pull request #9615: Mask other forms of password arguments in SparkSubmitOperator
turbaszek commented on a change in pull request #9615:
URL: https://github.com/apache/airflow/pull/9615#discussion_r449011845
##
File path: tests/providers/apache/spark/hooks/test_spark_submit.py
##
@@ -748,3 +750,64 @@ def test_k8s_process_on_kill(self, mock_popen,
mock_client_method):
client.delete_namespaced_pod.assert_called_once_with(
'spark-pi-edf2ace37be7353a958b38733a12f8e6-driver',
'mynamespace', **kwargs)
+
+
+@pytest.mark.parametrize(
+("command", "expected"),
+(
+(
+("spark-submit", "foo", "--bar", "baz", "--password='secret'"),
+"spark-submit foo --bar baz --password='**'",
+),
+(
+("spark-submit", "foo", "--bar", "baz", '--password="secret"'),
+'spark-submit foo --bar baz --password="**"',
+),
+(
+("spark-submit", "foo", "--bar", "baz", "--password=secret"),
+"spark-submit foo --bar baz --password=**",
+),
+(
+("spark-submit", "foo", "--bar", "baz", "--password 'secret'"),
+"spark-submit foo --bar baz --password '**'",
+),
+(
+("spark-submit", "foo", "--bar", "baz", "--password secret"),
+"spark-submit foo --bar baz --password **",
+),
+(
+("spark-submit", "foo", "--bar", "baz", '--password "secret"'),
+'spark-submit foo --bar baz --password "**"',
+),
+(
+("spark-submit", "foo", "--bar", "baz", "--secret='secret'"),
+"spark-submit foo --bar baz --secret='**'",
+),
+(
+("spark-submit", "foo", "--bar", "baz", "--foo.password='secret'"),
+"spark-submit foo --bar baz --foo.password='**'",
+),
+(
+("spark-submit",),
+"spark-submit",
+),
+
+(
+("spark-submit", "foo", "--bar", "baz", "--password \"secret'"),
+"spark-submit foo --bar baz --password \"secret'",
+),
+(
+("spark-submit", "foo", "--bar", "baz", "--password 'secret\""),
+"spark-submit foo --bar baz --password 'secret\"",
+),
+),
+)
+def test_masks_passwords(command: str, expected: str) -> None:
Review comment:
You can use https://pypi.org/project/parameterized/ in future we will
probably migrate to pytest
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9616: local job heartbeat callback should use session from provide_session
turbaszek commented on pull request #9616: URL: https://github.com/apache/airflow/pull/9616#issuecomment-653012832 @pingzh can you please take a look at the failing tests? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9623: Move ElasticsearchTaskHandler to the provider package
turbaszek commented on pull request #9623: URL: https://github.com/apache/airflow/pull/9623#issuecomment-653009423 @ephraimbuddy can you please take a look at the CI errors? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9624: Move StackdriverTaskHandler to the provider package
turbaszek commented on pull request #9624: URL: https://github.com/apache/airflow/pull/9624#issuecomment-653009381 @ephraimbuddy can you please take a look at the CI errors? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] j-y-matsubara commented on a change in pull request #9531: Support .airflowignore for plugins
j-y-matsubara commented on a change in pull request #9531: URL: https://github.com/apache/airflow/pull/9531#discussion_r448993794 ## File path: airflow/utils/file.py ## @@ -90,6 +90,47 @@ def open_maybe_zipped(fileloc, mode='r'): return io.open(fileloc, mode=mode) +def find_path_from_directory( +base_dir_path: str, +ignore_list_file: str) -> Generator[str, None, None]: +""" +Search the file and return the path of the file that should not be ignored. +:param base_dir_path: the base path to be searched for. +:param ignore_file_list_name: the file name in which specifies a regular expression pattern is written. Review comment: I'm sorry. It's my simple mistake. I fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] j-y-matsubara commented on a change in pull request #9531: Support .airflowignore for plugins
j-y-matsubara commented on a change in pull request #9531: URL: https://github.com/apache/airflow/pull/9531#discussion_r448993794 ## File path: airflow/utils/file.py ## @@ -90,6 +90,47 @@ def open_maybe_zipped(fileloc, mode='r'): return io.open(fileloc, mode=mode) +def find_path_from_directory( +base_dir_path: str, +ignore_list_file: str) -> Generator[str, None, None]: +""" +Search the file and return the path of the file that should not be ignored. +:param base_dir_path: the base path to be searched for. +:param ignore_file_list_name: the file name in which specifies a regular expression pattern is written. Review comment: I'm sorry. It's a simple mistake on my part. I fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9277: [WIP] Health endpoint spec
turbaszek commented on a change in pull request #9277: URL: https://github.com/apache/airflow/pull/9277#discussion_r448975124 ## File path: airflow/api_connexion/endpoints/health_endpoint.py ## @@ -14,13 +14,35 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. - -# TODO(mik-laj): We have to implement it. -# Do you want to help? Please look at: https://github.com/apache/airflow/issues/8144 +from airflow.api_connexion.schemas.health_schema import health_schema +from airflow.jobs.scheduler_job import SchedulerJob def get_health(): """ -Checks if the API works +Return the health of the airflow scheduler and metadatabase """ -return "OK" +HEALTHY = "healthy" # pylint: disable=invalid-name +UNHEALTHY = "unhealthy" # pylint: disable=invalid-name Review comment: If you will move those out of the function then there will be no need to disable pylint :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] OmairK commented on a change in pull request #9277: [WIP] Health endpoint spec
OmairK commented on a change in pull request #9277:
URL: https://github.com/apache/airflow/pull/9277#discussion_r448972246
##
File path: airflow/api_connexion/endpoints/health_endpoint.py
##
@@ -14,13 +14,34 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at:
https://github.com/apache/airflow/issues/8144
+from airflow.api_connexion.schemas.health_schema import health_schema
+from airflow.configuration import conf
+from airflow.jobs.scheduler_job import SchedulerJob
+from airflow.utils.session import provide_session
def get_health():
"""
-Checks if the API works
+Return the health of the airflow scheduler and metadatabase
"""
-return "OK"
+payload = {
+'metadatabase': {'status': 'unhealthy'}
+}
+
+latest_scheduler_heartbeat = None
+scheduler_status = 'unhealthy'
+payload['metadatabase'] = {'status': 'healthy'}
Review comment:
Thanks, fixed it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] OmairK commented on a change in pull request #9277: [WIP] Health endpoint spec
OmairK commented on a change in pull request #9277:
URL: https://github.com/apache/airflow/pull/9277#discussion_r448971809
##
File path: airflow/api_connexion/endpoints/health_endpoint.py
##
@@ -14,13 +14,34 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at:
https://github.com/apache/airflow/issues/8144
+from airflow.api_connexion.schemas.health_schema import health_schema
+from airflow.configuration import conf
+from airflow.jobs.scheduler_job import SchedulerJob
+from airflow.utils.session import provide_session
def get_health():
"""
-Checks if the API works
+Return the health of the airflow scheduler and metadatabase
"""
-return "OK"
+payload = {
+'metadatabase': {'status': 'unhealthy'}
+}
+
+latest_scheduler_heartbeat = None
+scheduler_status = 'unhealthy'
+payload['metadatabase'] = {'status': 'healthy'}
Review comment:
Thanks. [Fixed
it](https://github.com/apache/airflow/pull/9277/files#diff-d7bb321505e9703c67dc7c78ff5b55deR25-R48)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9593: Improve handling Dataproc cluster creation with ERROR state
turbaszek commented on pull request #9593: URL: https://github.com/apache/airflow/pull/9593#issuecomment-652980847 @olchas would you mind taking a look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #9627: MySQL support in Helm airflow database config
potiuk commented on issue #9627: URL: https://github.com/apache/airflow/issues/9627#issuecomment-652966722 Ach sorry. CC: @schnie (you are both just one line apart when typing "Greg" ). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9628: fix PR checks
potiuk commented on pull request #9628: URL: https://github.com/apache/airflow/pull/9628#issuecomment-652965696 I think this was a transient error - rerunning it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9593: Improve handling Dataproc cluster creation with ERROR state
turbaszek commented on pull request #9593: URL: https://github.com/apache/airflow/pull/9593#issuecomment-652954399 @dossett I've added handling of this case This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Dewsmen commented on issue #8557: Count SKIPPED as SUCCESS if wait_for_downstream=True
Dewsmen commented on issue #8557: URL: https://github.com/apache/airflow/issues/8557#issuecomment-652951751 The issue was fixed and merged with [PR](https://github.com/apache/airflow/pull/7735) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Dewsmen closed issue #8557: Count SKIPPED as SUCCESS if wait_for_downstream=True
Dewsmen closed issue #8557: URL: https://github.com/apache/airflow/issues/8557 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] aneesh-joseph opened a new pull request #9628: fix static checks
aneesh-joseph opened a new pull request #9628: URL: https://github.com/apache/airflow/pull/9628 --- Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Target Github ISSUE in description if exists - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] zikun edited a comment on issue #9609: TimeSensor triggers immediately when used over midnight (UTC)
zikun edited a comment on issue #9609:
URL: https://github.com/apache/airflow/issues/9609#issuecomment-652927309
I would say this is a feature request rather than a bug. The TimeSensor
behaves as expected.
I would like to propose a new sensor `DateTimeSensor`.
So in case above, it will be something like
```
sensor = DateTimeSensor(
task_id="sensor",
target_time='{{ next_execution_date.add(days=1).replace(hour=2,
minute=0) }}'
)
```
WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] zikun edited a comment on issue #9609: TimeSensor triggers immediately when used over midnight (UTC)
zikun edited a comment on issue #9609:
URL: https://github.com/apache/airflow/issues/9609#issuecomment-652927309
I would say this is a feature request rather than a bug. The TimeSensor
behaves as expected.
I would like to propose a new sensor `DateTimeSensor`.
So in case above, it will be something like
`time_04h00_local = DateTimeSensor(task_id="time_01h30", target_time='{{
next_execution_date }}.add(days=1).replace(hour=2, minute=0)`)
WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9472: Add drop_partition functionality for HiveMetastoreHook
turbaszek commented on a change in pull request #9472: URL: https://github.com/apache/airflow/pull/9472#discussion_r448915957 ## File path: tests/providers/apache/hive/hooks/test_hive.py ## @@ -383,6 +383,10 @@ def test_table_exists(self): self.hook.table_exists(str(random.randint(1, 1))) ) +def test_drop_partition(self): +self.assertTrue(self.hook.drop_partitions(self.table, db=self.database, + part_vals=[DEFAULT_DATE_DS])) + Review comment: Does this tests requires some external service? Does it create side effects? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9472: Add drop_partition functionality for HiveMetastoreHook
turbaszek commented on a change in pull request #9472: URL: https://github.com/apache/airflow/pull/9472#discussion_r448915752 ## File path: airflow/providers/apache/hive/hooks/hive.py ## @@ -775,6 +775,23 @@ def table_exists(self, table_name, db='default'): except Exception: # pylint: disable=broad-except return False +def drop_partitions(self, table_name, part_vals, delete_data=False, db='default'): +""" +Drop partitions matching param_names input +>>> hh = HiveMetastoreHook() +>>> hh.drop_partitions(db='airflow', table_name='static_babynames', +part_vals="['2020-05-01']") +True Review comment: Please add arguments information using `:param xxx:` and `:type xxx:` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9472: Add drop_partition functionality for HiveMetastoreHook
turbaszek commented on a change in pull request #9472: URL: https://github.com/apache/airflow/pull/9472#discussion_r448915531 ## File path: airflow/providers/apache/hive/hooks/hive.py ## @@ -775,6 +775,23 @@ def table_exists(self, table_name, db='default'): except Exception: # pylint: disable=broad-except return False +def drop_partitions(self, table_name, part_vals, delete_data=False, db='default'): Review comment: Please add type annotations This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch master updated (ce9bad4 -> ee20086)
This is an automated email from the ASF dual-hosted git repository.
turbaszek pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from ce9bad4 Improve queries number SchedulerJob._process_executor_events
(#9488)
add ee20086 Move S3TaskHandler to the AWS provider package (#9602)
No new revisions were added by this update.
Summary of changes:
UPDATING.md| 4 +
airflow/config_templates/airflow_local_settings.py | 2 +-
.../providers/amazon/aws/log}/__init__.py | 0
.../amazon/aws}/log/s3_task_handler.py | 0
airflow/utils/log/s3_task_handler.py | 177 ++---
docs/autoapi_templates/index.rst | 13 ++
.../{zendesk/hooks => amazon/aws/log}/__init__.py | 0
.../amazon/aws}/log/test_s3_task_handler.py| 6 +-
8 files changed, 33 insertions(+), 169 deletions(-)
copy {tests/providers/zendesk/hooks =>
airflow/providers/amazon/aws/log}/__init__.py (100%)
copy airflow/{utils => providers/amazon/aws}/log/s3_task_handler.py (100%)
copy tests/providers/{zendesk/hooks => amazon/aws/log}/__init__.py (100%)
rename tests/{utils => providers/amazon/aws}/log/test_s3_task_handler.py (96%)
[GitHub] [airflow] gstein commented on issue #9627: MySQL support in Helm airflow database config
gstein commented on issue #9627: URL: https://github.com/apache/airflow/issues/9627#issuecomment-652933545 Not sure why I was cc'd. ?? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek merged pull request #9602: Move S3TaskHandler to the provider package
turbaszek merged pull request #9602: URL: https://github.com/apache/airflow/pull/9602 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9277: [WIP] Health endpoint spec
turbaszek commented on a change in pull request #9277:
URL: https://github.com/apache/airflow/pull/9277#discussion_r448911785
##
File path: airflow/api_connexion/endpoints/health_endpoint.py
##
@@ -14,13 +14,34 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at:
https://github.com/apache/airflow/issues/8144
+from airflow.api_connexion.schemas.health_schema import health_schema
+from airflow.configuration import conf
+from airflow.jobs.scheduler_job import SchedulerJob
+from airflow.utils.session import provide_session
def get_health():
"""
-Checks if the API works
+Return the health of the airflow scheduler and metadatabase
"""
-return "OK"
+payload = {
+'metadatabase': {'status': 'unhealthy'}
+}
+
+latest_scheduler_heartbeat = None
+scheduler_status = 'unhealthy'
+payload['metadatabase'] = {'status': 'healthy'}
Review comment:
```python
latest_scheduler_heartbeat = None
scheduler_status = 'unhealthy'
metadatabase_status = 'healthy'
... # here update the statuses
payload = {
'metadatabase': {'status': metadatabase_status},
'scheduler': {
'status': scheduler_status,
'latest_scheduler_heartbeat': latest_scheduler_heartbeat,
}
}
return health_schema.dump(payload)
```
What do you think about this approach?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9277: [WIP] Health endpoint spec
turbaszek commented on a change in pull request #9277:
URL: https://github.com/apache/airflow/pull/9277#discussion_r448907827
##
File path: airflow/api_connexion/endpoints/health_endpoint.py
##
@@ -14,13 +14,34 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at:
https://github.com/apache/airflow/issues/8144
+from airflow.api_connexion.schemas.health_schema import health_schema
+from airflow.configuration import conf
+from airflow.jobs.scheduler_job import SchedulerJob
+from airflow.utils.session import provide_session
def get_health():
"""
-Checks if the API works
+Return the health of the airflow scheduler and metadatabase
"""
-return "OK"
+payload = {
+'metadatabase': {'status': 'unhealthy'}
+}
+
+latest_scheduler_heartbeat = None
+scheduler_status = 'unhealthy'
+payload['metadatabase'] = {'status': 'healthy'}
Review comment:
Let's use constants:
```
UNHEALTHY = "unhealthy"
HEALTHY = "healthy"
```
as we use those values in many places
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9277: [WIP] Health endpoint spec
turbaszek commented on a change in pull request #9277:
URL: https://github.com/apache/airflow/pull/9277#discussion_r448907827
##
File path: airflow/api_connexion/endpoints/health_endpoint.py
##
@@ -14,13 +14,34 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at:
https://github.com/apache/airflow/issues/8144
+from airflow.api_connexion.schemas.health_schema import health_schema
+from airflow.configuration import conf
+from airflow.jobs.scheduler_job import SchedulerJob
+from airflow.utils.session import provide_session
def get_health():
"""
-Checks if the API works
+Return the health of the airflow scheduler and metadatabase
"""
-return "OK"
+payload = {
+'metadatabase': {'status': 'unhealthy'}
+}
+
+latest_scheduler_heartbeat = None
+scheduler_status = 'unhealthy'
+payload['metadatabase'] = {'status': 'healthy'}
Review comment:
Let's use constants:
```
UNHEALTHY = "unhealthy"
HEALTHY = "healthy"
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[airflow] branch master updated (e50e946 -> ce9bad4)
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from e50e946 Task logging handlers can provide custom log links (#9354) add ce9bad4 Improve queries number SchedulerJob._process_executor_events (#9488) No new revisions were added by this update. Summary of changes: airflow/jobs/scheduler_job.py| 32 ++-- tests/jobs/test_scheduler_job.py | 4 ++-- 2 files changed, 20 insertions(+), 16 deletions(-)
[GitHub] [airflow] zikun commented on issue #9609: TimeSensor triggers immediately when used over midnight (UTC)
zikun commented on issue #9609:
URL: https://github.com/apache/airflow/issues/9609#issuecomment-652927309
I would say this is a feature request rather than a bug. The TimeSensor
behaves as expected.
I would like to propose a new sensor `DateTimeSensor`.
So in case above, it will be something like
`time_04h00_local = DateTimeSensor(task_id="time_01h30",
target_time=pendulum.parse('{{ next_execution_date
}}').add(days=1).replace(hour=2, minute=0)`
WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] turbaszek merged pull request #9488: Improve queries number SchedulerJob._process_executor_events
turbaszek merged pull request #9488: URL: https://github.com/apache/airflow/pull/9488 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9531: Support .airflowignore for plugins
turbaszek commented on a change in pull request #9531: URL: https://github.com/apache/airflow/pull/9531#discussion_r448905152 ## File path: airflow/utils/file.py ## @@ -90,6 +90,47 @@ def open_maybe_zipped(fileloc, mode='r'): return io.open(fileloc, mode=mode) +def find_path_from_directory( +base_dir_path: str, +ignore_list_file: str) -> Generator[str, None, None]: +""" +Search the file and return the path of the file that should not be ignored. +:param base_dir_path: the base path to be searched for. +:param ignore_file_list_name: the file name in which specifies a regular expression pattern is written. Review comment: ```suggestion ignore_file_name: str) -> Generator[str, None, None]: """ Search the file and return the path of the file that should not be ignored. :param base_dir_path: the base path to be searched for. :param ignore_file_name: the file name in which specifies a regular expression pattern is written. ``` WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #9627: MySQL support in Helm airflow database config
potiuk commented on issue #9627: URL: https://github.com/apache/airflow/issues/9627#issuecomment-652913931 It's not a bug, it's a feature. The current helm chart does not support MySQL but I agree it should. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #9627: MySQL support in Helm airflow database config
potiuk commented on issue #9627: URL: https://github.com/apache/airflow/issues/9627#issuecomment-652914104 cc: @dimberman @gstein This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (AIRFLOW-3909) cant read log file for previous tries with multiply celery workers
[
https://issues.apache.org/jira/browse/AIRFLOW-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150134#comment-17150134
]
Valeriys Soloviov edited comment on AIRFLOW-3909 at 7/2/20, 9:49 AM:
-
I am facing it on Airflow 1.10.3 and 1.10.9 with the following options:
all_dbs crypto async jdbc hdfs hive ldap mssql mysql password s3 slack
{code:java}
# lsb_release -a
Distributor ID: Debian
Description:Debian GNU/Linux 9.12 (stretch)
Release:9.12
Codename: stretch
# python3.6 --version
Python 3.6.9
{code}
[~ray.p...@gmail.com] / [~Trivedi] / [~mightydok] had you tried remote logging
to S3? Helped it?
was (Author: weldpua2008):
I am facing it on Airflow 1.10.3 and 1.10.9
[~ray.p...@gmail.com] / [~Trivedi] / [~mightydok] had you tried remote logging
to S3? Helped it?
> cant read log file for previous tries with multiply celery workers
> --
>
> Key: AIRFLOW-3909
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3909
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.2
>Reporter: Vitaliy Okulov
>Priority: Major
>
> With 1.10.2 version i have a error when try to read log via web interface for
> job that have multiply tries, and some of this tries executed on different
> celery worker than the first one.
> As example:
>
> {code:java}
> *** Log file does not exist:
> /usr/local/airflow/logs/php_firebot_log_2/php_firebot_log_task/2019-02-13T20:55:00+00:00/2.log
> *** Fetching from:
> http://airdafworker2:8793/log/php_firebot_log_2/php_firebot_log_task/2019-02-13T20:55:00+00:00/2.log
> *** Failed to fetch log file from worker. 404 Client Error: NOT FOUND for
> url:
> http://airdafworker2:8793/log/php_firebot_log_2/php_firebot_log_task/2019-02-13T20:55:00+00:00/2.log
> {code}
> But this task was executed on airdafworker1 worker, and log file exist on
> this host.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3909) cant read log file for previous tries with multiply celery workers
[
https://issues.apache.org/jira/browse/AIRFLOW-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150134#comment-17150134
]
Valeriys Soloviov commented on AIRFLOW-3909:
I am facing it on Airflow 1.10.3 and 1.10.9
[~ray.p...@gmail.com] / [~Trivedi] / [~mightydok] had you tried remote logging
to S3? Helped it?
> cant read log file for previous tries with multiply celery workers
> --
>
> Key: AIRFLOW-3909
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3909
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.2
>Reporter: Vitaliy Okulov
>Priority: Major
>
> With 1.10.2 version i have a error when try to read log via web interface for
> job that have multiply tries, and some of this tries executed on different
> celery worker than the first one.
> As example:
>
> {code:java}
> *** Log file does not exist:
> /usr/local/airflow/logs/php_firebot_log_2/php_firebot_log_task/2019-02-13T20:55:00+00:00/2.log
> *** Fetching from:
> http://airdafworker2:8793/log/php_firebot_log_2/php_firebot_log_task/2019-02-13T20:55:00+00:00/2.log
> *** Failed to fetch log file from worker. 404 Client Error: NOT FOUND for
> url:
> http://airdafworker2:8793/log/php_firebot_log_2/php_firebot_log_task/2019-02-13T20:55:00+00:00/2.log
> {code}
> But this task was executed on airdafworker1 worker, and log file exist on
> this host.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] boring-cyborg[bot] commented on issue #9627: MySQL support in Helm airflow database config
boring-cyborg[bot] commented on issue #9627: URL: https://github.com/apache/airflow/issues/9627#issuecomment-652903692 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Swalloow opened a new issue #9627: MySQL support in Helm airflow database config
Swalloow opened a new issue #9627: URL: https://github.com/apache/airflow/issues/9627 **Apache Airflow version**: 1.10.10 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.16 **Environment**: - **Cloud provider or hardware configuration**: AWS EKS - **OS** (e.g. from /etc/os-release): AWS EKS default AMI - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: airflow metadata db as AWS Aurora 5.6.4+ **What happened**: Deploy official helm chart from master branch and update `values.yaml` file as below. I use AWS Aurora as database. ``` # Airflow database config data: # If secret names are provided, use those secrets metadataSecretName: ~ resultBackendSecretName: ~ # Otherwise pass connection values in # FIXME: metadataConnection: user: MYAURORA_USER pass: MYAURORA_PASSWORD host: MYAURORA_URL port: 3306 db: airflow sslmode: disable resultBackendConnection: user: MYAURORA_USER pass: MYAURORA_PASSWORD host: MYAURORA_URL port: 3306 db: airflow sslmode: disable ``` After install helm chart, an error occurred in scheduler initContainer phase. ``` $ kubectl get po -n airflow NAME READY STATUS RESTARTS AGE airflow-dev-create-user-v84lr0/1 Error1 18s airflow-dev-scheduler-695894dbc7-jz9qx 0/2 Init:Error 1 18s airflow-dev-statsd-7f5fbdc8c4-49487 1/1 Running 0 18s airflow-dev-webserver-547664465f-hqh54 0/1 Init:0/1 0 18s $ kubectl describe po -n airflow airflow-dev-scheduler-695894dbc7-jz9qx ... Init Containers: run-airflow-migrations: Container ID: docker://9b431e2584a68afdcdacd7f7043e6facd46c38ab577dd427f14a3a79782a934d Image: apache/airflow:1.10.10.1-alpha2-python3.6 Image ID: docker-pullable://apache/airflow@sha256:f44d93b67f58a761aab9c4ed5804b4df2fd15050a9c561413d722eb619c064d5 Port: Host Port: Args: bash -c airflow upgradedb || airflow db upgrade State: Terminated Reason: Error Exit Code:2 Started: Thu, 02 Jul 2020 18:18:09 +0900 Finished: Thu, 02 Jul 2020 18:18:15 +0900 Last State: Terminated Reason: Error Exit Code:2 Started: Thu, 02 Jul 2020 18:17:23 +0900 Finished: Thu, 02 Jul 2020 18:17:29 +0900 Ready: False Restart Count: 4 Environment: AIRFLOW__CORE__FERNET_KEY:Optional: false AIRFLOW__CORE__SQL_ALCHEMY_CONN:Optional: false AIRFLOW_CONN_AIRFLOW_DB:Optional: false Mounts: /var/run/secrets/kubernetes.io/serviceaccount from airflow-dev-scheduler-serviceaccount-token-vh5td (ro) ``` **What you expected to happen**: I expected to support mysql addresses as well. However, if you look at the secret yaml file, it is fixed at db+postgresql:// address. (`chart/templates/secrets/result-backend-connection-secret.yaml`) **How to reproduce it**: helm install . --name airflow --namespace airflow **Anything else we need to know**: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (AIRFLOW-6062) Scheduler doesn't delete worker pods from namespaces different than the scheduler's namespace
[ https://issues.apache.org/jira/browse/AIRFLOW-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150129#comment-17150129 ] ASF GitHub Bot commented on AIRFLOW-6062: - mppetkov edited a comment on pull request #8546: URL: https://github.com/apache/airflow/pull/8546#issuecomment-652901414 One more vote for the generic multi-namespace approach, it would be amazing if we have functionality like that. Actually, that's why I have contributed the fix. In my company we need to run each worker Pod into a specific namespace based on a condition. From what I understand, Airflow needs to provide the following options in the **airflow.cfg** to make this approach generic. **- default_namespace: 'namespace1'** Airflow will create each worker Pod in this specific namespace unless the value is not overwritten. **- allowed_namespaces: ['\*']** Airflow can create a worker Pod in all namespaces (probably the whitelist property should have * as a value) **- allowed_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow can create a worker Pod in any namespace that is part of the **allowed_namespaces** property. **- denied_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow cannot create a worker Pod in any namespace that is part of the **denied_namespaces** property. What do you think about this approach? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Scheduler doesn't delete worker pods from namespaces different than the > scheduler's namespace > - > > Key: AIRFLOW-6062 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6062 > Project: Apache Airflow > Issue Type: Bug > Components: executor-kubernetes >Affects Versions: 1.10.5 >Reporter: Mihail Petkov >Assignee: Daniel Imberman >Priority: Blocker > Fix For: 1.10.10 > > > When you run Airflow's task instances as worker pods in different namespaces > into a Kubernetes cluster, the scheduler can delete only the pods that are > living in the same namespace where the scheduler lives. It's trying to delete > all pods that are in the namespace defined in the airflow.cfg file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6062) Scheduler doesn't delete worker pods from namespaces different than the scheduler's namespace
[ https://issues.apache.org/jira/browse/AIRFLOW-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150130#comment-17150130 ] ASF GitHub Bot commented on AIRFLOW-6062: - mppetkov edited a comment on pull request #8546: URL: https://github.com/apache/airflow/pull/8546#issuecomment-652901414 One more vote for the generic multi-namespace approach, it would be amazing if we have functionality like that. Actually, that's why I have contributed the fix. In my company we need to run each worker Pod into a specific namespace based on a condition. From what I understand, Airflow needs to provide the following options in the **airflow.cfg** to make this approach generic. **- default_namespace: 'namespace1'** Airflow will create each worker Pod in this specific namespace unless the value is not overwritten. **- allowed_namespaces: ['\*']** Airflow can create a worker Pod in all namespaces (probably the whitelist property should have * as a value) **- allowed_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow can create a worker Pod in any namespace that is part of the **allowed_namespaces** property. **- denied_namespaces: ['namespace4', 'namespace5', 'namespace6']** Airflow cannot create a worker Pod in any namespace that is part of the **denied_namespaces** property. What do you think about this approach? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Scheduler doesn't delete worker pods from namespaces different than the > scheduler's namespace > - > > Key: AIRFLOW-6062 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6062 > Project: Apache Airflow > Issue Type: Bug > Components: executor-kubernetes >Affects Versions: 1.10.5 >Reporter: Mihail Petkov >Assignee: Daniel Imberman >Priority: Blocker > Fix For: 1.10.10 > > > When you run Airflow's task instances as worker pods in different namespaces > into a Kubernetes cluster, the scheduler can delete only the pods that are > living in the same namespace where the scheduler lives. It's trying to delete > all pods that are in the namespace defined in the airflow.cfg file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mppetkov edited a comment on pull request #8546: AIRFLOW-6062 Watch worker pods from all namespaces
mppetkov edited a comment on pull request #8546: URL: https://github.com/apache/airflow/pull/8546#issuecomment-652901414 One more vote for the generic multi-namespace approach, it would be amazing if we have functionality like that. Actually, that's why I have contributed the fix. In my company we need to run each worker Pod into a specific namespace based on a condition. From what I understand, Airflow needs to provide the following options in the **airflow.cfg** to make this approach generic. **- default_namespace: 'namespace1'** Airflow will create each worker Pod in this specific namespace unless the value is not overwritten. **- allowed_namespaces: ['\*']** Airflow can create a worker Pod in all namespaces (probably the whitelist property should have * as a value) **- allowed_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow can create a worker Pod in any namespace that is part of the **allowed_namespaces** property. **- denied_namespaces: ['namespace4', 'namespace5', 'namespace6']** Airflow cannot create a worker Pod in any namespace that is part of the **denied_namespaces** property. What do you think about this approach? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mppetkov edited a comment on pull request #8546: AIRFLOW-6062 Watch worker pods from all namespaces
mppetkov edited a comment on pull request #8546: URL: https://github.com/apache/airflow/pull/8546#issuecomment-652901414 One more vote for the generic multi-namespace approach, it would be amazing if we have functionality like that. Actually, that's why I have contributed the fix. In my company we need to run each worker Pod into a specific namespace based on a condition. From what I understand, Airflow needs to provide the following options in the **airflow.cfg** to make this approach generic. **- default_namespace: 'namespace1'** Airflow will create each worker Pod in this specific namespace unless the value is not overwritten. **- allowed_namespaces: ['\*']** Airflow can create a worker Pod in all namespaces (probably the whitelist property should have * as a value) **- allowed_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow can create a worker Pod in any namespace that is part of the **allowed_namespaces** property. **- denied_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow cannot create a worker Pod in any namespace that is part of the **denied_namespaces** property. What do you think about this approach? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mppetkov edited a comment on pull request #8546: AIRFLOW-6062 Watch worker pods from all namespaces
mppetkov edited a comment on pull request #8546: URL: https://github.com/apache/airflow/pull/8546#issuecomment-652901414 One more vote for the generic multi-namespace approach, it would be amazing if we have functionality like that. Actually, that's why I have contributed the fix. In my company we need to run each worker Pod into a specific namespace based on a condition. From what I understand, Airflow needs to provide the following options in the **airflow.cfg** to make this approach generic. **- default_namespace: ['namespace1']** Airflow will create each worker Pod in this specific namespace unless the value is not overwritten. **- allowed_namespaces: ['\*']** Airflow can create a worker Pod in all namespaces (probably the whitelist property should have * as a value) **- allowed_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow can create a worker Pod in any namespace that is part of the **allowed_namespaces** property. **- denied_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow cannot create a worker Pod in any namespace that is part of the **denied_namespaces** property. What do you think about this approach? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (AIRFLOW-6062) Scheduler doesn't delete worker pods from namespaces different than the scheduler's namespace
[ https://issues.apache.org/jira/browse/AIRFLOW-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150128#comment-17150128 ] ASF GitHub Bot commented on AIRFLOW-6062: - mppetkov edited a comment on pull request #8546: URL: https://github.com/apache/airflow/pull/8546#issuecomment-652901414 One more vote for the generic multi-namespace approach, it would be amazing if we have functionality like that. Actually, that's why I have contributed the fix. In my company we need to run each worker Pod into a specific namespace based on a condition. From what I understand, Airflow needs to provide the following options in the **airflow.cfg** to make this approach generic. **- default_namespace: ['namespace1']** Airflow will create each worker Pod in this specific namespace unless the value is not overwritten. **- allowed_namespaces: ['\*']** Airflow can create a worker Pod in all namespaces (probably the whitelist property should have * as a value) **- allowed_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow can create a worker Pod in any namespace that is part of the **allowed_namespaces** property. **- denied_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow cannot create a worker Pod in any namespace that is part of the **denied_namespaces** property. What do you think about this approach? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Scheduler doesn't delete worker pods from namespaces different than the > scheduler's namespace > - > > Key: AIRFLOW-6062 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6062 > Project: Apache Airflow > Issue Type: Bug > Components: executor-kubernetes >Affects Versions: 1.10.5 >Reporter: Mihail Petkov >Assignee: Daniel Imberman >Priority: Blocker > Fix For: 1.10.10 > > > When you run Airflow's task instances as worker pods in different namespaces > into a Kubernetes cluster, the scheduler can delete only the pods that are > living in the same namespace where the scheduler lives. It's trying to delete > all pods that are in the namespace defined in the airflow.cfg file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mppetkov edited a comment on pull request #8546: AIRFLOW-6062 Watch worker pods from all namespaces
mppetkov edited a comment on pull request #8546: URL: https://github.com/apache/airflow/pull/8546#issuecomment-652901414 One more vote for the generic multi-namespace approach, it would be amazing if we have functionality like that. Actually, that's why I have contributed the fix. In my company we need to run each worker Pod into a specific namespace based on a condition. From what I understand, Airflow needs to provide the following options in the **airflow.cfg** to make this approach generic. **- allowed_namespaces: ['\*']** Airflow can create a worker Pod in all namespaces (probably the whitelist property should have * as a value) **- allowed_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow can create a worker Pod in any namespace that is part of the **allowed_namespaces** property. **- denied_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow cannot create a worker Pod in any namespace that is part of the **denied_namespaces** property. What do you think about this approach? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (AIRFLOW-6062) Scheduler doesn't delete worker pods from namespaces different than the scheduler's namespace
[ https://issues.apache.org/jira/browse/AIRFLOW-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150127#comment-17150127 ] ASF GitHub Bot commented on AIRFLOW-6062: - mppetkov edited a comment on pull request #8546: URL: https://github.com/apache/airflow/pull/8546#issuecomment-652901414 One more vote for the generic multi-namespace approach, it would be amazing if we have functionality like that. Actually, that's why I have contributed the fix. In my company we need to run each worker Pod into a specific namespace based on a condition. From what I understand, Airflow needs to provide the following options in the **airflow.cfg** to make this approach generic. **- allowed_namespaces: ['\*']** Airflow can create a worker Pod in all namespaces (probably the whitelist property should have * as a value) **- allowed_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow can create a worker Pod in any namespace that is part of the **allowed_namespaces** property. **- denied_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow cannot create a worker Pod in any namespace that is part of the **denied_namespaces** property. What do you think about this approach? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Scheduler doesn't delete worker pods from namespaces different than the > scheduler's namespace > - > > Key: AIRFLOW-6062 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6062 > Project: Apache Airflow > Issue Type: Bug > Components: executor-kubernetes >Affects Versions: 1.10.5 >Reporter: Mihail Petkov >Assignee: Daniel Imberman >Priority: Blocker > Fix For: 1.10.10 > > > When you run Airflow's task instances as worker pods in different namespaces > into a Kubernetes cluster, the scheduler can delete only the pods that are > living in the same namespace where the scheduler lives. It's trying to delete > all pods that are in the namespace defined in the airflow.cfg file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6062) Scheduler doesn't delete worker pods from namespaces different than the scheduler's namespace
[ https://issues.apache.org/jira/browse/AIRFLOW-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150126#comment-17150126 ] ASF GitHub Bot commented on AIRFLOW-6062: - mppetkov commented on pull request #8546: URL: https://github.com/apache/airflow/pull/8546#issuecomment-652901414 One more vote for the generic multi-namespace approach, it would be amazing if we have functionality like that. Actually, that's why I have contributed the fix. In my company we need to run each worker Pod into a specific namespace based on a condition. From what I understand, Airflow needs to provide the following options in the **airflow.cfg** to make this approach generic. **- allowed_namespaces: ['\*']** Airflow can create a worker Pod in all namespaces (probably the whitelist property should have * as a value) **- allowed_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow can create a worker Pod in any namespace that is part of the **allowed_namespaces** property. **- denied_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow cannot create a worker Pod in any namespace that is part of the **denied_namespaces** property. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Scheduler doesn't delete worker pods from namespaces different than the > scheduler's namespace > - > > Key: AIRFLOW-6062 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6062 > Project: Apache Airflow > Issue Type: Bug > Components: executor-kubernetes >Affects Versions: 1.10.5 >Reporter: Mihail Petkov >Assignee: Daniel Imberman >Priority: Blocker > Fix For: 1.10.10 > > > When you run Airflow's task instances as worker pods in different namespaces > into a Kubernetes cluster, the scheduler can delete only the pods that are > living in the same namespace where the scheduler lives. It's trying to delete > all pods that are in the namespace defined in the airflow.cfg file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on pull request #9531: Support .airflowignore for plugins
mik-laj commented on pull request #9531: URL: https://github.com/apache/airflow/pull/9531#issuecomment-652901310 Please ignore it. It is not related. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mppetkov commented on pull request #8546: AIRFLOW-6062 Watch worker pods from all namespaces
mppetkov commented on pull request #8546: URL: https://github.com/apache/airflow/pull/8546#issuecomment-652901414 One more vote for the generic multi-namespace approach, it would be amazing if we have functionality like that. Actually, that's why I have contributed the fix. In my company we need to run each worker Pod into a specific namespace based on a condition. From what I understand, Airflow needs to provide the following options in the **airflow.cfg** to make this approach generic. **- allowed_namespaces: ['\*']** Airflow can create a worker Pod in all namespaces (probably the whitelist property should have * as a value) **- allowed_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow can create a worker Pod in any namespace that is part of the **allowed_namespaces** property. **- denied_namespaces: ['namespace1', 'namespace2', 'namespace3']** Airflow cannot create a worker Pod in any namespace that is part of the **denied_namespaces** property. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] j-y-matsubara commented on pull request #9531: Support .airflowignore for plugins
j-y-matsubara commented on pull request #9531:
URL: https://github.com/apache/airflow/pull/9531#issuecomment-652898769
I think the two failure of k8s tests doesn't seem to have anything to
do with this PR. The same error occurs in other PRs. :(
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] zikun commented on issue #9626: Inconsistent logs
zikun commented on issue #9626: URL: https://github.com/apache/airflow/issues/9626#issuecomment-652893837 Thanks for the clarification. Understood. Will ping here to reopen it if I (or anyone else) find a clear way to reproduce the problem This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch master updated: Task logging handlers can provide custom log links (#9354)
This is an automated email from the ASF dual-hosted git repository.
turbaszek pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new e50e946 Task logging handlers can provide custom log links (#9354)
e50e946 is described below
commit e50e94613a671a86d72e687d9f27fe1cb73ebf36
Author: Mauricio De Diana
AuthorDate: Thu Jul 2 06:15:53 2020 -0300
Task logging handlers can provide custom log links (#9354)
Use a mixin to define log handlers based on remote services. The main
changes are:
- Create RemoteLoggingMixin to define remote log handlers.
- Remove explicit mentions to Elasticsearch in dag.html.
- Rename the /elasticsearch endpoint in views.py to
/redirect_to_remote_log and dispatch the remote URL building to the
log handler.
Co-authored-by: Kamil Breguła
---
airflow/config_templates/airflow_local_settings.py | 2 +
airflow/utils/log/es_task_handler.py | 34 +-
airflow/utils/log/log_reader.py| 8 ++-
airflow/utils/log/logging_mixin.py | 16 +
airflow/www/templates/airflow/dag.html | 30 +
airflow/www/views.py | 55
docs/howto/write-logs.rst | 26
tests/utils/log/test_es_task_handler.py| 22 +++
tests/www/test_views.py| 75 +-
9 files changed, 235 insertions(+), 33 deletions(-)
diff --git a/airflow/config_templates/airflow_local_settings.py
b/airflow/config_templates/airflow_local_settings.py
index 2374bd90..5633ea9 100644
--- a/airflow/config_templates/airflow_local_settings.py
+++ b/airflow/config_templates/airflow_local_settings.py
@@ -234,6 +234,7 @@ if REMOTE_LOGGING:
elif ELASTICSEARCH_HOST:
ELASTICSEARCH_LOG_ID_TEMPLATE: str = conf.get('elasticsearch',
'LOG_ID_TEMPLATE')
ELASTICSEARCH_END_OF_LOG_MARK: str = conf.get('elasticsearch',
'END_OF_LOG_MARK')
+ELASTICSEARCH_FRONTEND: str = conf.get('elasticsearch', 'frontend')
ELASTICSEARCH_WRITE_STDOUT: bool = conf.getboolean('elasticsearch',
'WRITE_STDOUT')
ELASTICSEARCH_JSON_FORMAT: bool = conf.getboolean('elasticsearch',
'JSON_FORMAT')
ELASTICSEARCH_JSON_FIELDS: str = conf.get('elasticsearch',
'JSON_FIELDS')
@@ -247,6 +248,7 @@ if REMOTE_LOGGING:
'filename_template': FILENAME_TEMPLATE,
'end_of_log_mark': ELASTICSEARCH_END_OF_LOG_MARK,
'host': ELASTICSEARCH_HOST,
+'frontend': ELASTICSEARCH_FRONTEND,
'write_stdout': ELASTICSEARCH_WRITE_STDOUT,
'json_format': ELASTICSEARCH_JSON_FORMAT,
'json_fields': ELASTICSEARCH_JSON_FIELDS
diff --git a/airflow/utils/log/es_task_handler.py
b/airflow/utils/log/es_task_handler.py
index 47f970f..b6e1687 100644
--- a/airflow/utils/log/es_task_handler.py
+++ b/airflow/utils/log/es_task_handler.py
@@ -18,6 +18,7 @@
import logging
import sys
+from urllib.parse import quote
# Using `from elasticsearch import *` would break elasticsearch mocking used
in unit test.
import elasticsearch
@@ -25,14 +26,15 @@ import pendulum
from elasticsearch_dsl import Search
from airflow.configuration import conf
+from airflow.models import TaskInstance
from airflow.utils import timezone
from airflow.utils.helpers import parse_template_string
from airflow.utils.log.file_task_handler import FileTaskHandler
from airflow.utils.log.json_formatter import JSONFormatter
-from airflow.utils.log.logging_mixin import LoggingMixin
+from airflow.utils.log.logging_mixin import ExternalLoggingMixin, LoggingMixin
-class ElasticsearchTaskHandler(FileTaskHandler, LoggingMixin):
+class ElasticsearchTaskHandler(FileTaskHandler, LoggingMixin,
ExternalLoggingMixin):
"""
ElasticsearchTaskHandler is a python log handler that
reads logs from Elasticsearch. Note logs are not directly
@@ -51,11 +53,13 @@ class ElasticsearchTaskHandler(FileTaskHandler,
LoggingMixin):
PAGE = 0
MAX_LINE_PER_PAGE = 1000
+LOG_NAME = 'Elasticsearch'
-def __init__(self, base_log_folder, filename_template,
+def __init__(self, base_log_folder, filename_template, # pylint:
disable=too-many-arguments
log_id_template, end_of_log_mark,
write_stdout, json_format, json_fields,
host='localhost:9200',
+ frontend='localhost:5601',
es_kwargs=conf.getsection("elasticsearch_configs")):
"""
:param base_log_folder: base folder to store logs locally
@@ -72,6 +76,7 @@ class ElasticsearchTaskHandler(FileTaskHandler, LoggingMixin):
self.client = elasticsearch.Elasticsearch([host], **es_kwargs)
+self.frontend = frontend
self.mark_end_on_c
[GitHub] [airflow] turbaszek merged pull request #9354: Task logging handlers can provide custom log links
turbaszek merged pull request #9354: URL: https://github.com/apache/airflow/pull/9354 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9502: generate go client from openapi spec
potiuk commented on a change in pull request #9502:
URL: https://github.com/apache/airflow/pull/9502#discussion_r448863282
##
File path: clients/gen/go.sh
##
@@ -0,0 +1,100 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+if [ "$#" -ne 2 ]; then
+echo "USAGE: $0 SPEC_PATH OUTPUT_DIR"
+exit 1
+fi
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+# shellcheck source=./clients/gen/common.sh
+source "${SCRIPT_DIR}/common.sh"
+
+VERSION=1.0.0
+go_config=(
+"packageVersion=${VERSION}"
+"enumClassPrefix=true"
+)
+
+SPEC_PATH=$(realpath "$1")
+if [ ! -d "$2" ]; then
+echo "$2 is not a valid directory or does not exist."
+exit 1
+fi
+OUTPUT_DIR=$(realpath "$2")
+
+# create openapi ignore file to keep generated code clean
+cat < "${OUTPUT_DIR}/.openapi-generator-ignore"
+.travis.yml
+git_push.sh
+EOF
+
+set -ex
+IFS=','
+
+SPEC_PATH="${SPEC_PATH}" \
+OUTPUT_DIR="${OUTPUT_DIR}" \
+gen_client go \
+--package-name airflow \
+--git-repo-id airflow/clients/go/airflow \
+--additional-properties "${go_config[*]}"
+
+# patch generated client to support problem HTTP API
+# this patch can be removed after following upstream patch gets merged:
+# https://github.com/OpenAPITools/openapi-generator/pull/6793
+cd "${OUTPUT_DIR}" && patch -b <<'EOF'
+--- client.go
client.go
+@@ -37,7 +37,7 @@ import (
+ )
+
+ var (
+- jsonCheck =
regexp.MustCompile(`(?i:(?:application|text)/(?:vnd\.[^;]+\+)?json)`)
++ jsonCheck =
regexp.MustCompile(`(?i:(?:application|text)/(?:vnd\.[^;]+\+)?(?:problem\+)?json)`)
+ xmlCheck = regexp.MustCompile(`(?i:(?:application|text)/xml)`)
+ )
+EOF
+
+# prepend apache license header
+cat < "./go-license-header"
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+EOF
+
+find "${OUTPUT_DIR}" -iname '*.go' \
+-exec \
+bash -c 'cat ./go-license-header $1 > $1.new && mv $1.new $1' _ {} \
+\;
+
+rm ./go-license-header
Review comment:
And we have pre-commit licence insert already working as pre-commit -
it's just a matter of copying some lines from
https://github.com/apache/airflow/blob/master/.pre-commit-config.yaml#L39 and
license template.
It also will work for all types of files - some files require different
headers etc.
It's a solved problem - let's just copy it from there.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] ashb closed issue #8683: fs_group and run_as_user is being defaulted to 0 while using kubernetes executor and airflow 1.10.10
ashb closed issue #8683: URL: https://github.com/apache/airflow/issues/8683 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on issue #8683: fs_group and run_as_user is being defaulted to 0 while using kubernetes executor and airflow 1.10.10
ashb commented on issue #8683: URL: https://github.com/apache/airflow/issues/8683#issuecomment-652888759 This has now been included in the v1-10-test branch as https://github.com/apache/airflow/commit/7ff7352a629a60106f6743257adad82367217c29 (commit SHA may yet change before release) -- closing this as it'll be fixed in 1.10.11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9502: generate go client from openapi spec
potiuk commented on a change in pull request #9502:
URL: https://github.com/apache/airflow/pull/9502#discussion_r448861326
##
File path: clients/gen/go.sh
##
@@ -0,0 +1,100 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+if [ "$#" -ne 2 ]; then
+echo "USAGE: $0 SPEC_PATH OUTPUT_DIR"
+exit 1
+fi
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+# shellcheck source=./clients/gen/common.sh
+source "${SCRIPT_DIR}/common.sh"
+
+VERSION=1.0.0
+go_config=(
+"packageVersion=${VERSION}"
+"enumClassPrefix=true"
+)
+
+SPEC_PATH=$(realpath "$1")
+if [ ! -d "$2" ]; then
+echo "$2 is not a valid directory or does not exist."
+exit 1
+fi
+OUTPUT_DIR=$(realpath "$2")
+
+# create openapi ignore file to keep generated code clean
+cat < "${OUTPUT_DIR}/.openapi-generator-ignore"
+.travis.yml
+git_push.sh
+EOF
+
+set -ex
+IFS=','
+
+SPEC_PATH="${SPEC_PATH}" \
+OUTPUT_DIR="${OUTPUT_DIR}" \
+gen_client go \
+--package-name airflow \
+--git-repo-id airflow/clients/go/airflow \
+--additional-properties "${go_config[*]}"
+
+# patch generated client to support problem HTTP API
+# this patch can be removed after following upstream patch gets merged:
+# https://github.com/OpenAPITools/openapi-generator/pull/6793
+cd "${OUTPUT_DIR}" && patch -b <<'EOF'
+--- client.go
client.go
+@@ -37,7 +37,7 @@ import (
+ )
+
+ var (
+- jsonCheck =
regexp.MustCompile(`(?i:(?:application|text)/(?:vnd\.[^;]+\+)?json)`)
++ jsonCheck =
regexp.MustCompile(`(?i:(?:application|text)/(?:vnd\.[^;]+\+)?(?:problem\+)?json)`)
+ xmlCheck = regexp.MustCompile(`(?i:(?:application|text)/xml)`)
+ )
+EOF
+
+# prepend apache license header
+cat < "./go-license-header"
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+EOF
+
+find "${OUTPUT_DIR}" -iname '*.go' \
+-exec \
+bash -c 'cat ./go-license-header $1 > $1.new && mv $1.new $1' _ {} \
+\;
+
+rm ./go-license-header
Review comment:
running script as pre-commit is super-easy and good practice.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9502: generate go client from openapi spec
mik-laj commented on a change in pull request #9502:
URL: https://github.com/apache/airflow/pull/9502#discussion_r448857057
##
File path: clients/gen/go.sh
##
@@ -0,0 +1,100 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+if [ "$#" -ne 2 ]; then
+echo "USAGE: $0 SPEC_PATH OUTPUT_DIR"
+exit 1
+fi
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+# shellcheck source=./clients/gen/common.sh
+source "${SCRIPT_DIR}/common.sh"
+
+VERSION=1.0.0
+go_config=(
+"packageVersion=${VERSION}"
+"enumClassPrefix=true"
+)
+
+SPEC_PATH=$(realpath "$1")
+if [ ! -d "$2" ]; then
+echo "$2 is not a valid directory or does not exist."
+exit 1
+fi
+OUTPUT_DIR=$(realpath "$2")
+
+# create openapi ignore file to keep generated code clean
+cat < "${OUTPUT_DIR}/.openapi-generator-ignore"
+.travis.yml
+git_push.sh
+EOF
+
+set -ex
+IFS=','
+
+SPEC_PATH="${SPEC_PATH}" \
+OUTPUT_DIR="${OUTPUT_DIR}" \
+gen_client go \
+--package-name airflow \
+--git-repo-id airflow/clients/go/airflow \
+--additional-properties "${go_config[*]}"
+
+# patch generated client to support problem HTTP API
+# this patch can be removed after following upstream patch gets merged:
+# https://github.com/OpenAPITools/openapi-generator/pull/6793
+cd "${OUTPUT_DIR}" && patch -b <<'EOF'
+--- client.go
client.go
+@@ -37,7 +37,7 @@ import (
+ )
+
+ var (
+- jsonCheck =
regexp.MustCompile(`(?i:(?:application|text)/(?:vnd\.[^;]+\+)?json)`)
++ jsonCheck =
regexp.MustCompile(`(?i:(?:application|text)/(?:vnd\.[^;]+\+)?(?:problem\+)?json)`)
+ xmlCheck = regexp.MustCompile(`(?i:(?:application|text)/xml)`)
+ )
+EOF
+
+# prepend apache license header
+cat < "./go-license-header"
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+EOF
+
+find "${OUTPUT_DIR}" -iname '*.go' \
+-exec \
+bash -c 'cat ./go-license-header $1 > $1.new && mv $1.new $1' _ {} \
+\;
+
+rm ./go-license-header
Review comment:
I don't know if it won't be too much engineering. We have a simple goal
- to add text at the beginning of the file. We can do it with a simple bash
script, or add pre-commit and its dependency as a dependency to this script
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #9622: Fix docstrings in exceptions.py
kaxil merged pull request #9622: URL: https://github.com/apache/airflow/pull/9622 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch master updated (cd3d9d9 -> 5cf2585)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from cd3d9d9 Fix using .json template extension in GMP operators (#9566) add 5cf2585 Fix docstrings in exceptions.py (#9622) No new revisions were added by this update. Summary of changes: airflow/exceptions.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [airflow] turbaszek commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
turbaszek commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r448848371
##
File path: airflow/providers/google/cloud/operators/bigquery.py
##
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+def _submit_job(self, hook: BigQueryHook, job_id: str):
+# Submit a new job
+job = hook.insert_job(
+configuration=self.configuration,
+project_id=self.project_id,
+location=self.location,
+job_id=job_id,
+)
+# Start the job and wait for it to complete and get the result.
+job.result()
+return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
-job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+exec_date = context['execution_date'].isoformat()
+job_id = self.job_id or
f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
Review comment:
This works in case we want to reattach to running job or job that
succeded. The case of rerunning failed task is handled in L1727
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9590: Improve idempotency of BigQueryInsertJobOperator
turbaszek commented on a change in pull request #9590:
URL: https://github.com/apache/airflow/pull/9590#discussion_r448847105
##
File path: airflow/providers/google/cloud/operators/bigquery.py
##
@@ -1692,32 +1692,52 @@ def prepare_template(self) -> None:
with open(self.configuration, 'r') as file:
self.configuration = json.loads(file.read())
+def _submit_job(self, hook: BigQueryHook, job_id: str):
+# Submit a new job
+job = hook.insert_job(
+configuration=self.configuration,
+project_id=self.project_id,
+location=self.location,
+job_id=job_id,
+)
+# Start the job and wait for it to complete and get the result.
+job.result()
+return job
+
def execute(self, context: Any):
hook = BigQueryHook(
gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
)
-job_id = self.job_id or f"airflow_{self.task_id}_{int(time())}"
+exec_date = context['execution_date'].isoformat()
+job_id = self.job_id or
f"airflow_{self.dag_id}_{self.task_id}_{exec_date}"
+
try:
-job = hook.insert_job(
-configuration=self.configuration,
-project_id=self.project_id,
-location=self.location,
-job_id=job_id,
-)
-# Start the job and wait for it to complete and get the result.
-job.result()
+# Submit a new job
+job = self._submit_job(hook, job_id)
except Conflict:
+# If the job already exists retrieve it
job = hook.get_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
)
-# Get existing job and wait for it to be ready
-for time_to_wait in exponential_sleep_generator(initial=10,
maximum=120):
-sleep(time_to_wait)
-job.reload()
-if job.done():
-break
+
+if job.done() and job.error_result:
+# The job exists and finished with an error and we are
probably reruning it
+# So we have to make a new job_id because it has to be unique
+job_id = f"{self.job_id}_{int(time())}"
+job = self._submit_job(hook, job_id)
+elif not job.done():
+# The job is still running so wait for it to be ready
+for time_to_wait in exponential_sleep_generator(initial=10,
maximum=120):
Review comment:
Can you provide the code? I tried to use the `job.result()` but it
doesn't work.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[airflow] branch master updated: Fix using .json template extension in GMP operators (#9566)
This is an automated email from the ASF dual-hosted git repository.
turbaszek pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new cd3d9d9 Fix using .json template extension in GMP operators (#9566)
cd3d9d9 is described below
commit cd3d9d93402f06a08f35e3586802f11a18c4f1f3
Author: Tomek Urbaszek
AuthorDate: Thu Jul 2 10:43:44 2020 +0200
Fix using .json template extension in GMP operators (#9566)
---
.../marketing_platform/operators/campaign_manager.py | 8 +++-
.../marketing_platform/operators/display_video.py | 7 +++
.../google/marketing_platform/operators/search_ads.py | 7 +++
docs/howto/operator/gcp/campaign_manager.rst | 3 ++-
docs/howto/operator/gcp/display_video.rst | 3 ++-
docs/howto/operator/gcp/search_ads.rst| 3 ++-
.../operators/test_campaign_manager.py| 19 +++
.../operators/test_display_video.py | 15 +++
.../marketing_platform/operators/test_search_ads.py | 19 +--
9 files changed, 78 insertions(+), 6 deletions(-)
diff --git
a/airflow/providers/google/marketing_platform/operators/campaign_manager.py
b/airflow/providers/google/marketing_platform/operators/campaign_manager.py
index 4814c9a..b3cd0c8 100644
--- a/airflow/providers/google/marketing_platform/operators/campaign_manager.py
+++ b/airflow/providers/google/marketing_platform/operators/campaign_manager.py
@@ -18,6 +18,7 @@
"""
This module contains Google CampaignManager operators.
"""
+import json
import tempfile
import uuid
from typing import Any, Dict, List, Optional
@@ -298,6 +299,12 @@ class
GoogleCampaignManagerInsertReportOperator(BaseOperator):
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
+def prepare_template(self) -> None:
+# If .json is passed then we have to read the file
+if isinstance(self.report, str) and self.report.endswith('.json'):
+with open(self.report, 'r') as file:
+self.report = json.load(file)
+
def execute(self, context: Dict):
hook = GoogleCampaignManagerHook(
gcp_conn_id=self.gcp_conn_id,
@@ -349,7 +356,6 @@ class GoogleCampaignManagerRunReportOperator(BaseOperator):
"gcp_conn_id",
"delegate_to",
)
-template_ext = (".json",)
@apply_defaults
def __init__(
diff --git
a/airflow/providers/google/marketing_platform/operators/display_video.py
b/airflow/providers/google/marketing_platform/operators/display_video.py
index 0b0af20..8f5f5b4 100644
--- a/airflow/providers/google/marketing_platform/operators/display_video.py
+++ b/airflow/providers/google/marketing_platform/operators/display_video.py
@@ -19,6 +19,7 @@
This module contains Google DisplayVideo operators.
"""
import csv
+import json
import shutil
import tempfile
import urllib.request
@@ -75,6 +76,12 @@ class
GoogleDisplayVideo360CreateReportOperator(BaseOperator):
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
+def prepare_template(self) -> None:
+# If .json is passed then we have to read the file
+if isinstance(self.body, str) and self.body.endswith('.json'):
+with open(self.body, 'r') as file:
+self.body = json.load(file)
+
def execute(self, context: Dict):
hook = GoogleDisplayVideo360Hook(
gcp_conn_id=self.gcp_conn_id,
diff --git
a/airflow/providers/google/marketing_platform/operators/search_ads.py
b/airflow/providers/google/marketing_platform/operators/search_ads.py
index ef88315..4f2200f 100644
--- a/airflow/providers/google/marketing_platform/operators/search_ads.py
+++ b/airflow/providers/google/marketing_platform/operators/search_ads.py
@@ -18,6 +18,7 @@
"""
This module contains Google Search Ads operators.
"""
+import json
from tempfile import NamedTemporaryFile
from typing import Any, Dict, Optional
@@ -70,6 +71,12 @@ class GoogleSearchAdsInsertReportOperator(BaseOperator):
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
+def prepare_template(self) -> None:
+# If .json is passed then we have to read the file
+if isinstance(self.report, str) and self.report.endswith('.json'):
+with open(self.report, 'r') as file:
+self.report = json.load(file)
+
def execute(self, context: Dict):
hook = GoogleSearchAdsHook(
gcp_conn_id=self.gcp_conn_id,
diff --git a/docs/howto/operator/gcp/campaign_manager.rst
b/docs/howto/operator/gcp/campaign_manager.rst
index a82de69..80b3b90 100644
--- a/docs/howto/operator/gcp/campaign_manager.rst
+++ b/docs/howto/operator/gcp/campaign_manager.rst
@@ -104,7 +104,8 @@ Running this operator creates a new report.
You can use :ref:`Jinja templating ` with
:template-
[GitHub] [airflow] turbaszek closed issue #9541: GoogleCampaignManagerInsertReportOperator Fails when using Templated Reports
turbaszek closed issue #9541: URL: https://github.com/apache/airflow/issues/9541 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek merged pull request #9566: Fix using .json template extension in GMP operators
turbaszek merged pull request #9566: URL: https://github.com/apache/airflow/pull/9566 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #9502: generate go client from openapi spec
feluelle commented on a change in pull request #9502:
URL: https://github.com/apache/airflow/pull/9502#discussion_r448838699
##
File path: clients/gen/go.sh
##
@@ -0,0 +1,100 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+if [ "$#" -ne 2 ]; then
+echo "USAGE: $0 SPEC_PATH OUTPUT_DIR"
+exit 1
+fi
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+# shellcheck source=./clients/gen/common.sh
+source "${SCRIPT_DIR}/common.sh"
+
+VERSION=1.0.0
+go_config=(
+"packageVersion=${VERSION}"
+"enumClassPrefix=true"
+)
+
+SPEC_PATH=$(realpath "$1")
+if [ ! -d "$2" ]; then
+echo "$2 is not a valid directory or does not exist."
+exit 1
+fi
+OUTPUT_DIR=$(realpath "$2")
+
+# create openapi ignore file to keep generated code clean
+cat < "${OUTPUT_DIR}/.openapi-generator-ignore"
+.travis.yml
+git_push.sh
+EOF
+
+set -ex
+IFS=','
+
+SPEC_PATH="${SPEC_PATH}" \
+OUTPUT_DIR="${OUTPUT_DIR}" \
+gen_client go \
+--package-name airflow \
+--git-repo-id airflow/clients/go/airflow \
+--additional-properties "${go_config[*]}"
+
+# patch generated client to support problem HTTP API
+# this patch can be removed after following upstream patch gets merged:
+# https://github.com/OpenAPITools/openapi-generator/pull/6793
+cd "${OUTPUT_DIR}" && patch -b <<'EOF'
+--- client.go
client.go
+@@ -37,7 +37,7 @@ import (
+ )
+
+ var (
+- jsonCheck =
regexp.MustCompile(`(?i:(?:application|text)/(?:vnd\.[^;]+\+)?json)`)
++ jsonCheck =
regexp.MustCompile(`(?i:(?:application|text)/(?:vnd\.[^;]+\+)?(?:problem\+)?json)`)
+ xmlCheck = regexp.MustCompile(`(?i:(?:application|text)/xml)`)
+ )
+EOF
+
+# prepend apache license header
+cat < "./go-license-header"
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+EOF
+
+find "${OUTPUT_DIR}" -iname '*.go' \
+-exec \
+bash -c 'cat ./go-license-header $1 > $1.new && mv $1.new $1' _ {} \
+\;
+
+rm ./go-license-header
Review comment:
Can we do this via a pre-commit hook instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9556: API Endpoint - DagRuns Batch
turbaszek commented on a change in pull request #9556: URL: https://github.com/apache/airflow/pull/9556#discussion_r448835585 ## File path: tests/api_connexion/endpoints/test_dag_run_endpoint.py ## @@ -346,6 +346,243 @@ def test_end_date_gte_lte(self, url, expected_dag_run_ids, session): self.assertEqual(dag_run_ids, expected_dag_run_ids) +class TestGetDagRunBatch(TestDagRunEndpoint): +@provide_session +def test_should_respond_200(self, session): +dag_runs = self._create_test_dag_run() +session.add_all(dag_runs) +session.commit() Review comment: Or maybe it would be better to extend `_create_test_dag_run` with commit option? For example `_create_test_dag_run(commit=True)` so we can avoid code duplication This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9556: API Endpoint - DagRuns Batch
turbaszek commented on a change in pull request #9556:
URL: https://github.com/apache/airflow/pull/9556#discussion_r448834084
##
File path: tests/api_connexion/endpoints/test_dag_run_endpoint.py
##
@@ -346,6 +346,243 @@ def test_end_date_gte_lte(self, url,
expected_dag_run_ids, session):
self.assertEqual(dag_run_ids, expected_dag_run_ids)
+class TestGetDagRunBatch(TestDagRunEndpoint):
+@provide_session
+def test_should_respond_200(self, session):
+dag_runs = self._create_test_dag_run()
+session.add_all(dag_runs)
+session.commit()
+payload = {
+"dag_ids": ["TEST_DAG_ID"],
+}
+response = self.client.post("api/v1/dags/~/dagRuns/list", json=payload)
+assert response.status_code == 200
+self.assertEqual(
Review comment:
Should we use the same style of asserts in test? I would opt for `assert
x == y`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #9556: API Endpoint - DagRuns Batch
turbaszek commented on a change in pull request #9556: URL: https://github.com/apache/airflow/pull/9556#discussion_r448833504 ## File path: tests/api_connexion/endpoints/test_dag_run_endpoint.py ## @@ -346,6 +346,243 @@ def test_end_date_gte_lte(self, url, expected_dag_run_ids, session): self.assertEqual(dag_run_ids, expected_dag_run_ids) +class TestGetDagRunBatch(TestDagRunEndpoint): +@provide_session +def test_should_respond_200(self, session): +dag_runs = self._create_test_dag_run() +session.add_all(dag_runs) +session.commit() Review comment: ```suggestion def test_should_respond_200(self): dag_runs = self._create_test_dag_run() with create_session() as session: session.add_all(dag_runs) ``` WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #9604: Move CloudwatchTaskHandler to the provider package
feluelle commented on a change in pull request #9604: URL: https://github.com/apache/airflow/pull/9604#discussion_r448832881 ## File path: UPDATING.md ## @@ -61,6 +61,9 @@ More tips can be found in the guide: https://developers.google.com/style/inclusive-documentation --> +### CloudwatchTaskHandler have moved +The `CloudwatchTaskHandler` class from `airflow.utils.log.cloudwatch_task_handler` has been moved to +`airflow.providers.amazon.aws.log.cloudwatch_task_handler`. This is because it has items specific to `aws` Review comment: ```suggestion `airflow.providers.amazon.aws.log.cloudwatch_task_handler`. This is because it has items specific to `aws`. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #9626: Inconsistent logs
potiuk commented on issue #9626: URL: https://github.com/apache/airflow/issues/9626#issuecomment-652858744 The problem is that this issue has not enough information to be able to reproduce it. This is really the case "let's find out what's going on - but simply without your feedback and other people commenting and providing their details - this issue here is not actionable. Once a least the root cause and easily reproducible path on how to reproduce it is there and we clearly know it's a bug and not misconfiguration - we made the decision that those kind of issues should not be added in Github issues. At this stage it's more "question or support" rather than "bug' IMHO (unless I am mistaken of course). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9556: API Endpoint - DagRuns Batch
ephraimbuddy commented on a change in pull request #9556: URL: https://github.com/apache/airflow/pull/9556#discussion_r448810142 ## File path: airflow/api_connexion/schemas/dag_run_schema.py ## @@ -72,5 +72,25 @@ class DAGRunCollectionSchema(Schema): total_entries = fields.Int() +class DagRunsBatchFormSchema(Schema): +""" Schema to validate and deserialize the Form(request payload) submitted to DagRun Batch endpoint""" + +class Meta: +""" Meta """ +datetimeformat = 'iso' +strict = True + +page_offset = fields.Int(required=False, missing=0, min=0) +page_limit = fields.Int(required=False, missing=100, min=1) Review comment: I think it's better left out. Some servers can handle more than 1000 page request limit. I'm working on configurable page request limit and we can apply it here. Then it'll not be a hard limit but a limit that the users can configure. You can see the commit here https://github.com/apache/airflow/commit/a37ac22c179e5dccc95b05c862bffc542dad125e This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9277: [WIP] Health endpoint spec
mik-laj commented on pull request #9277: URL: https://github.com/apache/airflow/pull/9277#issuecomment-652839628 @turbaszek Can you look at it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] zikun commented on issue #9626: Inconsistent logs
zikun commented on issue #9626: URL: https://github.com/apache/airflow/issues/9626#issuecomment-652839098 I suspect it's a systematic bug, not a specific situation to troubleshoot because I see another person reported a similar bug in slack - https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1593033265401600 Anyway, I'll also report it in slack. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (AIRFLOW-6372) Align Azure Blob Storage remote logging URI scheme with S3 and GCS
[ https://issues.apache.org/jira/browse/AIRFLOW-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150020#comment-17150020 ] ASF GitHub Bot commented on AIRFLOW-6372: - stale[bot] commented on pull request #6931: URL: https://github.com/apache/airflow/pull/6931#issuecomment-652838388 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Align Azure Blob Storage remote logging URI scheme with S3 and GCS > -- > > Key: AIRFLOW-6372 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6372 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Affects Versions: 1.10.6 >Reporter: Yuen-Kuei Hsueh >Assignee: Yuen-Kuei Hsueh >Priority: Minor > Fix For: 2.0.0 > > > The `remote_base_log_folder` for S3 and GCS are URI Schemes, but Azure Blob > Storage only contains path and [requires additional setting for container > name|https://airflow.apache.org/docs/stable/howto/write-logs.html#writing-logs-to-azure-blob-storage]. > For Hadoop ecosystem, the [URI Scheme for Azure Blob > Storage|https://hadoop.apache.org/docs/current/hadoop-azure/index.html#Accessing_wasb_URLs] > was `wasb[s]://@.blob.core.windows.net/`. > So I think we can use Hadoop URI Scheme for Azure Blob Storage to align the > configuration and make `wasb_container` parameter optional. > The changes should be also compatible with legacy configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6794) Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
[
https://issues.apache.org/jira/browse/AIRFLOW-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150019#comment-17150019
]
ASF GitHub Bot commented on AIRFLOW-6794:
-
stale[bot] commented on pull request #7688:
URL: https://github.com/apache/airflow/pull/7688#issuecomment-652838412
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs. Thank you for
your contributions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
> -
>
> Key: AIRFLOW-6794
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6794
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, operators
>Affects Versions: 1.10.9
>Reporter: Roger Russel Droique Neris
>Assignee: Omkar
>Priority: Trivial
> Labels: AWS, newbie
>
> {{The Redshift operator }}
> {{"airflow.providers.amazon.aws.operators.redshift_to_s3.}}{{RedshiftToS3Transfer"
> allow only a simple usage to transfer a table to a S3. }}
> {{[https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/operators/redshift_to_s3.py#L110]}}
> {{If possible I would like to implement an usage of a custom query on it.}}
> {{The behavior expected is when a "query" parameter is given then it wil use
> it, and if the it was not given it will use the default behavior.}}
> {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] stale[bot] commented on pull request #8002: DST bug #7999 failing unit test
stale[bot] commented on pull request #8002: URL: https://github.com/apache/airflow/pull/8002#issuecomment-652838397 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] stale[bot] commented on pull request #6931: [AIRFLOW-6372] Align WASB remote logging URI scheme
stale[bot] commented on pull request #6931: URL: https://github.com/apache/airflow/pull/6931#issuecomment-652838388 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] stale[bot] commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
stale[bot] commented on pull request #7688: URL: https://github.com/apache/airflow/pull/7688#issuecomment-652838412 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] stale[bot] commented on pull request #8870: Add template_fields to GCSFileTransformOperator
stale[bot] commented on pull request #8870: URL: https://github.com/apache/airflow/pull/8870#issuecomment-652838400 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] OmairK commented on a change in pull request #9556: API Endpoint - DagRuns Batch
OmairK commented on a change in pull request #9556: URL: https://github.com/apache/airflow/pull/9556#discussion_r448800768 ## File path: airflow/api_connexion/schemas/dag_run_schema.py ## @@ -72,5 +72,25 @@ class DAGRunCollectionSchema(Schema): total_entries = fields.Int() +class DagRunsBatchFormSchema(Schema): +""" Schema to validate and deserialize the Form(request payload) submitted to DagRun Batch endpoint""" + +class Meta: +""" Meta """ +datetimeformat = 'iso' +strict = True + +page_offset = fields.Int(required=False, missing=0, min=0) +page_limit = fields.Int(required=False, missing=100, min=1) Review comment: ```suggestion page_limit = fields.Int(required=False, missing=100, min=1, max=100) ``` It would be better to cap the limit to 100. @ephraimbuddy @takunnithan what do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #9626: Inconsistent logs
potiuk commented on issue #9626: URL: https://github.com/apache/airflow/issues/9626#issuecomment-652836297 Hello @zikun. This issue is not enough to be treated as a bug report - it's more troubleshooting that you can take to our slack channel. See https://airflow.apache.org/community/ " ask a question" chapter. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk closed issue #9626: Inconsistent logs
potiuk closed issue #9626: URL: https://github.com/apache/airflow/issues/9626 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] VinayGb665 commented on a change in pull request #9549: YAML file supports extra json parameters
VinayGb665 commented on a change in pull request #9549: URL: https://github.com/apache/airflow/pull/9549#discussion_r448802216 ## File path: docs/howto/use-alternative-secrets-backend.rst ## @@ -118,6 +123,7 @@ The following is a sample JSON file. The YAML file structure is similar to that of a JSON. The key-value pair of connection ID and the definitions of one or more connections. The connection can be defined as a URI (string) or JSON object. + For a guide about defining a connection as a URI, see:: :ref:`generating_connection_uri`. For a description of the connection object parameters see :class:`~airflow.models.connection.Connection`. Review comment: Dropped This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] VinayGb665 commented on a change in pull request #9549: YAML file supports extra json parameters
VinayGb665 commented on a change in pull request #9549: URL: https://github.com/apache/airflow/pull/9549#discussion_r448802150 ## File path: docs/howto/use-alternative-secrets-backend.rst ## @@ -92,8 +92,13 @@ file ``/files/my_conn.json`` when it looks for connections. The file can be defined in ``JSON``, ``YAML`` or ``env`` format. +Any extra json parameters can be provided using keys like ``extra_dejson`` and ``extra``. +The key ``extra_dejson`` can be used to provide parameters as JSON object where as the key ``extra`` can be used in case of a JSON string. +The keys ``extra`` and ``extra_dejson`` are mutually exclusive. + The JSON file must contain an object where the key contains the connection ID and the value contains the definitions of one or more connections. The connection can be defined as a URI (string) or JSON object. + For a guide about defining a connection as a URI, see:: :ref:`generating_connection_uri`. For a description of the connection object parameters see :class:`~airflow.models.connection.Connection`. The following is a sample JSON file. Review comment: Moved it to the inro This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] zikun opened a new issue #9626: Inconsistent logs
zikun opened a new issue #9626:
URL: https://github.com/apache/airflow/issues/9626
**Apache Airflow version**: 1.10.10
**Environment**:
Official Docker image
Python 3.7
CeleryExecutor with worker_autoscale = 512,16
Scheduler, worker and webserver running in three different Docker containers.
- **Cloud provider or hardware configuration**:
AWS EC2
- **OS** (e.g. from /etc/os-release):
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/";
SUPPORT_URL="https://www.debian.org/support";
BUG_REPORT_URL="https://bugs.debian.org/";
- **Kernel** (e.g. `uname -a`):
Linux dumm.hostname.com 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43
UTC 2018 x86_64 GNU/Linux
- **Install tools**:
git
procps
OpenJDK JRE 11 headless
- **Others**:
pip packages
jaydebeapi==1.2.0
docker==4.2.1
**What happened**:
There are two problems and they seem to always appear together.
* The logs for running tasks are not consistent. They are overwritten during
task runs.
* Some `FAILED: Task is in the 'running' state which is not a valid state
for execution. The task must be cleared in order to be run.` messages in the
log but eventually the task succeeds.
Below are some snapshots from a `JDBCOperator` task log:
Task Log
## First snapshot
airflow@xxx:/opt/airflow$ cat
logs/dag_id/task_id/2020-07-01T22:07:00+00:00/4.log
[2020-07-02 05:59:24,226] {taskinstance.py:669} INFO - Dependencies all met
for
[2020-07-02 05:59:24,248] {taskinstance.py:669} INFO - Dependencies all met
for
[2020-07-02 05:59:24,248] {taskinstance.py:879} INFO -
[2020-07-02 05:59:24,248] {taskinstance.py:880} INFO - Starting attempt 4 of
5
[2020-07-02 05:59:24,248] {taskinstance.py:881} INFO -
[2020-07-02 05:59:24,267] {taskinstance.py:900} INFO - Executing
on 2020-07-01T22:07:00+00:00
[2020-07-02 05:59:24,270] {standard_task_runner.py:53} INFO - Started
process 16712 to run task
[2020-07-02 05:59:24,358] {logging_mixin.py:112} INFO - Running %s on host
%s xxx
[2020-07-02 05:59:24,410] {jdbc_operator.py:61} INFO - Executing: ['CREATE
OR REPLACE TABLE TEST.T1 AS (SELECT * FROM TEST.DUMMY WHERE ID <= 10;',
'SELECT * FROM TEST.T1 ORDER BY DBTIMESTAMP DESC limit 10;', 'CREATE OR REPLACE
TABLE TEST.T2 AS (SELECT * FROM TEST.DUMMY WHERE ID > 10;', 'SELECT * FROM
TEST.T2 limit 10;']
[2020-07-02 05:59:24,415] {logging_mixin.py:112} INFO - [2020-07-02
05:59:24,415] {base_hook.py:87} INFO - Using connection to: id: exasol_prod_rw.
Host: jdbc:exa:172.18.229.31..47:8563;schema=EXDDATA, Port: None, Schema: ,
Login: exddata, Password: , extra:
[2020-07-02 05:59:25,396] {logging_mixin.py:112} INFO - [2020-07-02
05:59:25,396] {dbapi_hook.py:174} INFO - CREATE OR REPLACE TABLE TEST.T1 AS
(SELECT * FROM TEST.DUMMY WHERE ID <= 10;
## Second snapshot
airflow@xxx:/opt/airflow$ cat
logs/dag_id/task_id/2020-07-01T22:07:00+00:00/4.log
[2020-07-02 06:00:55,465] {taskinstance.py:663} INFO - Dependencies not met
for ,
dependency 'Task Instance State' FAILED: Task is in the 'running' state which
is not a valid state for execution. The task must be cleared in order to be run.
[2020-07-02 06:00:55,468] {logging_mixin.py:112} INFO - [2020-07-02
06:00:55,468] {local_task_job.py:91} INFO - Task is not able to be run
[2020-07-02 06:01:20,667] {logging_mixin.py:112} INFO - [2020-07-02
06:01:20,667] {dbapi_hook.py:174} INFO - SELECT * FROM TEST.T2 limit 10;
## Third snapshot
airflow@xxx:/opt/airflow$ cat
logs/dag_id/task_id/2020-07-01T22:07:00+00:00/4.log
[2020-07-02 06:00:55,465] {taskinstance.py:663} INFO - Dependencies not met
for ,
dependency 'Task Instance State' FAILED: Task is in the 'running' state which
is not a valid state for execution. The task must be cleared in order to be run.
[2020-07-02 06:00:55,468] {logging_mixin.py:112} INFO - [2020-07-02
06:00:55,468] {local_task_job.py:91} INFO - Task is not able to be run
[2020-07-02 06:01:20,667] {logging_mixin.py:112} INFO - [2020-07-02
06:01:20,667] {dbapi_hook.py:174} INFO - SELECT * FROM TEST.T2 limit 10;
[2020-07-02 06:01:48,119] {taskinstance.py:1065} INFO - Marking task as
SUCCESS.dag_id=dag_id, task_id=task_id, execution_date=20200701T220700,
start_date=20200702T055924, end_date=20200702T060148
[2020-07-02 06:01:50,254] {logging_mixin.py:112} INFO - [2020-07-02
06:01:50,254] {local_task_job.py:103} INFO - Task exited with return code 0
**What you expected to happen**:
The log should not be appended not overwritten.
[GitHub] [airflow] mik-laj commented on a change in pull request #9549: YAML file supports extra json parameters
mik-laj commented on a change in pull request #9549: URL: https://github.com/apache/airflow/pull/9549#discussion_r448790018 ## File path: docs/howto/use-alternative-secrets-backend.rst ## @@ -118,6 +123,7 @@ The following is a sample JSON file. The YAML file structure is similar to that of a JSON. The key-value pair of connection ID and the definitions of one or more connections. The connection can be defined as a URI (string) or JSON object. + For a guide about defining a connection as a URI, see:: :ref:`generating_connection_uri`. For a description of the connection object parameters see :class:`~airflow.models.connection.Connection`. Review comment: This content is unnecessarily duplicated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9549: YAML file supports extra json parameters
mik-laj commented on a change in pull request #9549: URL: https://github.com/apache/airflow/pull/9549#discussion_r448789877 ## File path: docs/howto/use-alternative-secrets-backend.rst ## @@ -92,8 +92,13 @@ file ``/files/my_conn.json`` when it looks for connections. The file can be defined in ``JSON``, ``YAML`` or ``env`` format. +Any extra json parameters can be provided using keys like ``extra_dejson`` and ``extra``. +The key ``extra_dejson`` can be used to provide parameters as JSON object where as the key ``extra`` can be used in case of a JSON string. +The keys ``extra`` and ``extra_dejson`` are mutually exclusive. + The JSON file must contain an object where the key contains the connection ID and the value contains the definitions of one or more connections. The connection can be defined as a URI (string) or JSON object. + For a guide about defining a connection as a URI, see:: :ref:`generating_connection_uri`. For a description of the connection object parameters see :class:`~airflow.models.connection.Connection`. The following is a sample JSON file. Review comment: You can move this sentence for introduction. We have the first use of this concept there, so these links should be earlier. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
