date:20221218



dstandish opened a new pull request, #28454:
URL: https://github.com/apache/airflow/pull/28454

   We used to pickle raw pods objects but found that when unpickling across k8s 
lib versions we would get missing attr errors.  We can fix this by 
roundtripping to json in a more tolerant fashion than is done by the 
openapi-generated code, i.e. by populating missing attrs with None.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28453: Update dates.py

boring-cyborg[bot] commented on PR #28453:
URL: https://github.com/apache/airflow/pull/28453#issuecomment-1357193944

Congratulations on your first Pull Request and welcome to the Apache Airflow
community! If you have any issues or are unsure about any anything please check
our Contribution Guide
(https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type
annotations). Our [pre-commits](
https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
will help you with that.
- In case of a new feature add useful documentation (in docstrings or in
`docs/` directory). Adding a new operator? Check this short
[guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst)
Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze
environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for
testing locally, it's a heavy docker but it ships with a working Airflow and a
lot of integrations.
- Be patient and persistent. It might take some time to get a review or get
the final approval from Committers.
- Please follow [ASF Code of
Conduct](https://www.apache.org/foundation/policies/conduct) for all
communication including (but not limited to) comments on Pull Requests, Mailing
list and Slack.
- Be sure to read the [Airflow Coding style](
https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it
better .
In case of doubts contact the developers at:
Mailing List: d...@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] xyc-pz opened a new pull request, #28453: Update dates.py



xyc-pz opened a new pull request, #28453:
URL: https://github.com/apache/airflow/pull/28453

   The example import sentence in line 51  is incorrect
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Soonmok commented on a diff in pull request #28400: Rewrite timezone assign codeline



Soonmok commented on code in PR #28400:
URL: https://github.com/apache/airflow/pull/28400#discussion_r1051854266


##
airflow/models/dag.py:
##
@@ -462,8 +462,10 @@ def __init__(
 # set timezone from start_date
 tz = None
 if start_date and start_date.tzinfo:
-tzinfo = None if start_date.tzinfo else settings.TIMEZONE
-tz = pendulum.instance(start_date, tz=tzinfo).timezone
+if start_date.tzinfo is None:
+tz = pendulum.instance(start_date, 
tz=settings.TIMEZONE).timezone
+else:
+tz = pendulum.instance(start_date).timezone

Review Comment:
   ah, right. I made a mistake. 
   Should be 
   ```
   if start_date:
   tzinfo = None if start_date.tzinfo else settings.TIMEZONE
   tz = pendulum.instance(start_date, tz=tzinfo).timezone
   if start_date.tzinfo is None:
   tz = pendulum.instance(start_date, tz=settings.TIMEZONE).timezone
   else:
   tz = pendulum.instance(start_date).timezone
   ```
   but I think we don't need to proceed with this change. 
   I will look another issues up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Soonmok closed pull request #28400: Rewrite timezone assign codeline



Soonmok closed pull request #28400: Rewrite timezone assign codeline
URL: https://github.com/apache/airflow/pull/28400


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Add `ensure_ascii=False` in trigger dag run API (#28451)

2022-12-18 Thread ephraimanierobi

This is an automated email from the ASF dual-hosted git repository.

ephraimanierobi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new c3eee43725 Add `ensure_ascii=False` in trigger dag run API (#28451)
c3eee43725 is described below

commit c3eee4372556f9b09d3395a3f251c9ee21278846
Author: Mansu Kim 
AuthorDate: Mon Dec 19 15:09:36 2022 +0900

Add `ensure_ascii=False` in trigger dag run API (#28451)

* Add ensure_ascii=False in trigger dag run api

* Fix static checks
---
 airflow/www/views.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/airflow/www/views.py b/airflow/www/views.py
index fe40cf25c3..baf806b5f5 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -1992,7 +1992,9 @@ class Airflow(AirflowBaseView):
 else:
 try:
 default_conf = json.dumps(
-{str(k): v.resolve(suppress_exception=True) for k, v 
in dag.params.items()}, indent=4
+{str(k): v.resolve(suppress_exception=True) for k, v 
in dag.params.items()},
+indent=4,
+ensure_ascii=False,
 )
 except TypeError:
 flash("Could not pre-populate conf field due to 
non-JSON-serializable data-types")

[GitHub] [airflow] ephraimbuddy merged pull request #28451: Add `ensure_ascii=False` in trigger dag run API



ephraimbuddy merged PR #28451:
URL: https://github.com/apache/airflow/pull/28451


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a diff in pull request #28436: Log FileTaskHandler to work with multi_namespace_mode when running using KubernetesExecutor



dstandish commented on code in PR #28436:
URL: https://github.com/apache/airflow/pull/28436#discussion_r1051830952


##
airflow/utils/log/file_task_handler.py:
##
@@ -191,16 +191,20 @@ def _read(self, ti: TaskInstance, try_number: int, 
metadata: dict[str, Any] | No
 log += f"*** {str(e)}\n"
 return log, {"end_of_log": True}
 elif self._should_check_k8s(ti.queue):
+pod_override = ti.executor_config.get("pod_override")
+if pod_override and pod_override.metadata and 
pod_override.metadata.namespace:
+namespace = pod_override.metadata.namespace
+else:
+namespace = conf.get("kubernetes_executor", "namespace")

Review Comment:
   yes, true.
   
   > All the database that Airflow is compatible with do support JSON column
   
   yeah we have solution for the "db does not support json", we just... dump to 
json string.  we have a number of fields like this now.
   
   > if it will mean harder to write efficient SQL/SQLAlchemy queries.
   
   yeah but that's only an issue if you are going to need to filter on that 
field.  my assumption is, we'd only be looking at this field after we already 
have the TI loaded.  similar to executor config, which is even worse since it's 
a binary blob field (pickled obj) -- but here too it's no problem since we 
never do queries based on that field.
   
   anyway, no emergency to add this... just ... something we might wanna do
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bharanidharan14 commented on a diff in pull request #28262: Hook for managing directories and files in Azure Data Lake Storage Gen2



bharanidharan14 commented on code in PR #28262:
URL: https://github.com/apache/airflow/pull/28262#discussion_r1051823169


##
airflow/providers/microsoft/azure/hooks/adls.py:
##
@@ -0,0 +1,221 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import Any
+
+from azure.core.exceptions import ResourceExistsError, ResourceNotFoundError
+from azure.identity import ClientSecretCredential
+from azure.storage.filedatalake import (
+DataLakeDirectoryClient,
+DataLakeFileClient,
+DataLakeServiceClient,
+FileSystemClient,
+)
+
+from airflow.hooks.base import BaseHook
+
+
+class AzureDataLakeStorageV2(BaseHook):
+
+conn_name_attr = "adls_v2_conn_id"
+default_conn_name = "adls_v2_default"
+conn_type = "adls_v2"
+hook_name = "Azure Date Lake Storage"
+
+@staticmethod
+def get_connection_form_widgets() -> dict[str, Any]:
+"""Returns connection widgets to add to connection form"""
+from flask_appbuilder.fieldwidgets import BS3PasswordFieldWidget, 
BS3TextFieldWidget
+from flask_babel import lazy_gettext
+from wtforms import PasswordField, StringField
+
+return {
+"extra__adls_v2__connection_string": PasswordField(
+lazy_gettext("Blob Storage Connection String (optional)"), 
widget=BS3PasswordFieldWidget()
+),
+"extra__adls_v2__tenant_id": StringField(
+lazy_gettext("Tenant Id (Active Directory Auth)"), 
widget=BS3TextFieldWidget()
+),
+}
+
+@staticmethod
+def get_ui_field_behaviour() -> dict[str, Any]:
+"""Returns custom field behaviour"""
+return {
+"hidden_fields": ["schema", "port"],
+"relabeling": {
+"login": "Blob Storage Login (optional)",
+"password": "Blob Storage Key (optional)",
+"host": "Account Name (Active Directory Auth)",
+},
+"placeholders": {
+"login": "account name",
+"password": "secret",
+"host": "account url",
+"extra__adls_v2__connection_string": "connection string auth",
+"extra__adls_v2__tenant_id": "tenant",
+},
+}
+
+def __init__(self, adls_v2_conn_id: str = default_conn_name, public_read: 
bool = False) -> None:
+super().__init__()
+self.conn_id = adls_v2_conn_id
+self.public_read = public_read
+self.service_client = self.get_conn()
+
+def get_conn(self) -> DataLakeServiceClient:
+"""Return the DataLakeServiceClient object."""
+conn = self.get_connection(self.conn_id)
+extra = conn.extra_dejson or {}
+
+connection_string = extra.pop(
+"connection_string", 
extra.pop("extra__adls_v2__connection_string", None)
+)
+if connection_string:
+# connection_string auth takes priority
+return 
DataLakeServiceClient.from_connection_string(connection_string, **extra)
+
+tenant = extra.pop("tenant_id", extra.pop("extra__adls_v2__tenant_id", 
None))
+if tenant:
+# use Active Directory auth
+app_id = conn.login
+app_secret = conn.password
+token_credential = ClientSecretCredential(tenant, app_id, 
app_secret)
+return DataLakeServiceClient(
+account_url=f"https://{conn.login}.dfs.core.windows.net;, 
credential=token_credential, **extra
+)
+credential = conn.password

Review Comment:
   @tatiana ADLS gen2 is using different protocols and APIs and different 
client-type principles to connect. Currently, I have created separate 
connection types and connection details.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bharanidharan14 commented on a diff in pull request #28262: Hook for managing directories and files in Azure Data Lake Storage Gen2



bharanidharan14 commented on code in PR #28262:
URL: https://github.com/apache/airflow/pull/28262#discussion_r1051823169


##
airflow/providers/microsoft/azure/hooks/adls.py:
##
@@ -0,0 +1,221 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import Any
+
+from azure.core.exceptions import ResourceExistsError, ResourceNotFoundError
+from azure.identity import ClientSecretCredential
+from azure.storage.filedatalake import (
+DataLakeDirectoryClient,
+DataLakeFileClient,
+DataLakeServiceClient,
+FileSystemClient,
+)
+
+from airflow.hooks.base import BaseHook
+
+
+class AzureDataLakeStorageV2(BaseHook):
+
+conn_name_attr = "adls_v2_conn_id"
+default_conn_name = "adls_v2_default"
+conn_type = "adls_v2"
+hook_name = "Azure Date Lake Storage"
+
+@staticmethod
+def get_connection_form_widgets() -> dict[str, Any]:
+"""Returns connection widgets to add to connection form"""
+from flask_appbuilder.fieldwidgets import BS3PasswordFieldWidget, 
BS3TextFieldWidget
+from flask_babel import lazy_gettext
+from wtforms import PasswordField, StringField
+
+return {
+"extra__adls_v2__connection_string": PasswordField(
+lazy_gettext("Blob Storage Connection String (optional)"), 
widget=BS3PasswordFieldWidget()
+),
+"extra__adls_v2__tenant_id": StringField(
+lazy_gettext("Tenant Id (Active Directory Auth)"), 
widget=BS3TextFieldWidget()
+),
+}
+
+@staticmethod
+def get_ui_field_behaviour() -> dict[str, Any]:
+"""Returns custom field behaviour"""
+return {
+"hidden_fields": ["schema", "port"],
+"relabeling": {
+"login": "Blob Storage Login (optional)",
+"password": "Blob Storage Key (optional)",
+"host": "Account Name (Active Directory Auth)",
+},
+"placeholders": {
+"login": "account name",
+"password": "secret",
+"host": "account url",
+"extra__adls_v2__connection_string": "connection string auth",
+"extra__adls_v2__tenant_id": "tenant",
+},
+}
+
+def __init__(self, adls_v2_conn_id: str = default_conn_name, public_read: 
bool = False) -> None:
+super().__init__()
+self.conn_id = adls_v2_conn_id
+self.public_read = public_read
+self.service_client = self.get_conn()
+
+def get_conn(self) -> DataLakeServiceClient:
+"""Return the DataLakeServiceClient object."""
+conn = self.get_connection(self.conn_id)
+extra = conn.extra_dejson or {}
+
+connection_string = extra.pop(
+"connection_string", 
extra.pop("extra__adls_v2__connection_string", None)
+)
+if connection_string:
+# connection_string auth takes priority
+return 
DataLakeServiceClient.from_connection_string(connection_string, **extra)
+
+tenant = extra.pop("tenant_id", extra.pop("extra__adls_v2__tenant_id", 
None))
+if tenant:
+# use Active Directory auth
+app_id = conn.login
+app_secret = conn.password
+token_credential = ClientSecretCredential(tenant, app_id, 
app_secret)
+return DataLakeServiceClient(
+account_url=f"https://{conn.login}.dfs.core.windows.net;, 
credential=token_credential, **extra
+)
+credential = conn.password

Review Comment:
   @tatiana ADLS gen2 is using different protocols and APIs to connect. 
Currently I have created separate connection type and connection details.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bharanidharan14 commented on a diff in pull request #28262: Hook for managing directories and files in Azure Data Lake Storage Gen2



bharanidharan14 commented on code in PR #28262:
URL: https://github.com/apache/airflow/pull/28262#discussion_r1051820566


##
airflow/providers/microsoft/azure/hooks/adls.py:
##
@@ -0,0 +1,264 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import Any
+
+from azure.core.exceptions import ResourceExistsError, ResourceNotFoundError
+from azure.identity import ClientSecretCredential
+from azure.storage.filedatalake import (
+DataLakeDirectoryClient,
+DataLakeFileClient,
+DataLakeServiceClient,
+DirectoryProperties,
+FileSystemClient,
+FileSystemProperties,
+)
+
+from airflow.providers.microsoft.azure.hooks.wasb import WasbHook
+
+
+class AzureDataLakeStorageClient(WasbHook):

Review Comment:
   @luanmorenomaciel I don't think inheriting from the 
**airflow.providers.microsoft.azure.hooks.azure_data_lake** is also not 
recommended because this ADLS existing hook is gen1 and it's being getting 
retired from feb. They may stop supporting for that too as well. so its better 
to inherit from `BaseHook`. WDYT ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a diff in pull request #28436: Log FileTaskHandler to work with multi_namespace_mode when running using KubernetesExecutor



XD-DENG commented on code in PR #28436:
URL: https://github.com/apache/airflow/pull/28436#discussion_r1051814838


##
airflow/utils/log/file_task_handler.py:
##
@@ -191,16 +191,20 @@ def _read(self, ti: TaskInstance, try_number: int, 
metadata: dict[str, Any] | No
 log += f"*** {str(e)}\n"
 return log, {"end_of_log": True}
 elif self._should_check_k8s(ti.queue):
+pod_override = ti.executor_config.get("pod_override")
+if pod_override and pod_override.metadata and 
pod_override.metadata.namespace:
+namespace = pod_override.metadata.namespace
+else:
+namespace = conf.get("kubernetes_executor", "namespace")

Review Comment:
   I do agree that `Kubernetes` being "defacto" for Airflow (in very near 
future if not now). But it may not necessarily mean `KubernetesExecutor` going 
to be the "defacto" for Airflow? E.g. running `CeleryExecutor` (but on 
Kubernetes) still has certain pros over `KubernetesExecutor`.
   
   The idea to use a json column sounds good to me, meanwhile a few thoughts 
about it:
   - All the database that Airflow is compatible with do support JSON column, 
but not all database do that (so if later we want to add compatibility with 
another database, this may pose constraint).
   - if it will mean harder to write efficient SQL/SQLAlchemy queries.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on a diff in pull request #28400: Rewrite timezone assign codeline



uranusjr commented on code in PR #28400:
URL: https://github.com/apache/airflow/pull/28400#discussion_r1051813105


##
airflow/models/dag.py:
##
@@ -462,8 +462,10 @@ def __init__(
 # set timezone from start_date
 tz = None
 if start_date and start_date.tzinfo:
-tzinfo = None if start_date.tzinfo else settings.TIMEZONE
-tz = pendulum.instance(start_date, tz=tzinfo).timezone
+if start_date.tzinfo is None:
+tz = pendulum.instance(start_date, 
tz=settings.TIMEZONE).timezone
+else:
+tz = pendulum.instance(start_date).timezone

Review Comment:
   This seems wrong? The outer `if if start_date and start_date.tzinfo` should 
already ensure `start_date.tzinfo` is not None.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #28451: Add `ensure_ascii=False` in trigger dag run API



uranusjr commented on PR #28451:
URL: https://github.com/apache/airflow/pull/28451#issuecomment-1357108752

   Please fix the static check error.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'



dstandish commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1357099778

   The problem is that when you take a kubernetes object created from one lib 
version, and pickle it, then when you unpickle within a python env with a diff 
kubernetes lib version, you may get an object that will not work properly, and 
may fail when repr is called.
   
   This first appeared with executor_config pod_override pod objects.  We also 
saw it in db callback request objects.
   
   The solution was to run through airflow's serde before  / after pickling 
(executor config is stored as pickle object)
   
   You were are on 2.2.5.  So your executor_config objects are raw pickle 
objects.  You also had an older version of k8s.  Now you have a new k8s version 
and the objects are getting unpickled and blowing up.  
   
   Our solution makes things stable on a go forward basis but doesn't go back 
and "fix" the executor configs pickled with the old k8s version (we can't 
practically do so because we can't unpickle them properly in the new k8s 
version).  I mean, probably we could do some immense hackery to fix the issue 
but it's really a k8s lib problem, just an unfortunate consequence of the way 
that the kubernetes objects behave with pickle, and in practice this should 
only appear for a few TI's shortly after upgrade.
   
   You may try clearing the offending TI / nulling out the executor config.  
   
   But... i will take another look at the report / stacktrace to see if the 
scheduler was actually blowing up because, it shouldn't be doing that.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch constraints-main updated: Updating constraints. Build id:

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new 44b2632f0d Updating constraints. Build id:
44b2632f0d is described below

commit 44b2632f0d57d33f80ebbde82ff12268e5173d07
Author: Automated GitHub Actions commit 
AuthorDate: Mon Dec 19 04:43:01 2022 +

Updating constraints. Build id:

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
HEAD of '' in ''
with commit sha .

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.10.txt  | 4 ++--
 constraints-3.7.txt   | 4 ++--
 constraints-3.8.txt   | 4 ++--
 constraints-3.9.txt   | 4 ++--
 constraints-no-providers-3.10.txt | 2 +-
 constraints-no-providers-3.7.txt  | 2 +-
 constraints-no-providers-3.8.txt  | 2 +-
 constraints-no-providers-3.9.txt  | 2 +-
 constraints-source-providers-3.10.txt | 4 ++--
 constraints-source-providers-3.7.txt  | 4 ++--
 constraints-source-providers-3.8.txt  | 4 ++--
 constraints-source-providers-3.9.txt  | 4 ++--
 12 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/constraints-3.10.txt b/constraints-3.10.txt
index 55c380a306..00452b83f8 100644
--- a/constraints-3.10.txt
+++ b/constraints-3.10.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-19T02:01:59Z
+# This constraints file was automatically generated on 2022-12-19T04:42:14Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -593,7 +593,7 @@ types-paramiko==2.12.0.1
 types-protobuf==4.21.0.2
 types-python-dateutil==2.8.19.5
 types-python-slugify==7.0.0.1
-types-pytz==2022.6.0.1
+types-pytz==2022.7.0.0
 types-redis==4.3.21.6
 types-requests==2.28.11.5
 types-setuptools==65.6.0.2
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 8fde8e0f6c..7c5d02f73a 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-19T02:02:19Z
+# This constraints file was automatically generated on 2022-12-19T04:42:57Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -594,7 +594,7 @@ types-paramiko==2.12.0.1
 types-protobuf==4.21.0.2
 types-python-dateutil==2.8.19.5
 types-python-slugify==7.0.0.1
-types-pytz==2022.6.0.1
+types-pytz==2022.7.0.0
 types-redis==4.3.21.6
 types-requests==2.28.11.5
 types-setuptools==65.6.0.2
diff --git a/constraints-3.8.txt b/constraints-3.8.txt
index 3440a43687..1f2bd4477c 100644
--- a/constraints-3.8.txt
+++ b/constraints-3.8.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-19T02:02:11Z
+# This constraints file was automatically generated on 2022-12-19T04:42:44Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -596,7 +596,7 @@ types-paramiko==2.12.0.1
 types-protobuf==4.21.0.2
 types-python-dateutil==2.8.19.5
 types-python-slugify==7.0.0.1
-types-pytz==2022.6.0.1
+types-pytz==2022.7.0.0
 types-redis==4.3.21.6
 types-requests==2.28.11.5
 types-setuptools==65.6.0.2
diff --git a/constraints-3.9.txt b/constraints-3.9.txt
index ce1facccb3..da011a5672 100644
--- a/constraints-3.9.txt
+++ b/constraints-3.9.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-19T02:02:10Z
+# This constraints file was automatically generated on 2022-12-19T04:42:42Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -593,7 +593,7 @@ types-paramiko==2.12.0.1
 types-protobuf==4.21.0.2
 types-python-dateutil==2.8.19.5
 types-python-slugify==7.0.0.1
-types-pytz==2022.6.0.1
+types-pytz==2022.7.0.0
 types-redis==4.3.21.6
 types-requests==2.28.11.5
 types-setuptools==65.6.0.2
diff --git a/constraints-no-providers-3.10.txt 
b/constraints-no-providers-3.10.txt
index

[GitHub] [airflow] dstandish commented on a diff in pull request #28436: Log FileTaskHandler to work with multi_namespace_mode when running using KubernetesExecutor



dstandish commented on code in PR #28436:
URL: https://github.com/apache/airflow/pull/28436#discussion_r1051781510


##
airflow/utils/log/file_task_handler.py:
##
@@ -191,16 +191,20 @@ def _read(self, ti: TaskInstance, try_number: int, 
metadata: dict[str, Any] | No
 log += f"*** {str(e)}\n"
 return log, {"end_of_log": True}
 elif self._should_check_k8s(ti.queue):
+pod_override = ti.executor_config.get("pod_override")
+if pod_override and pod_override.metadata and 
pod_override.metadata.namespace:
+namespace = pod_override.metadata.namespace
+else:
+namespace = conf.get("kubernetes_executor", "namespace")

Review Comment:
   Yeah that occurred to me... And I think i probably agree... couple 
thoughts...
   
   For one, there is precedent.  e.g. `external_executor_id` and `queue` are 
columns on TI which only* apply for celery executor. (with an asterisk for 
queue; its role was expanded a little recently for LKE and CKE).  But, 
precedent isn't everything of course.
   
   And, kubernetes is, I believe, the defacto way to run airlfow, and more and 
more it's not either k8s executor OR other executors --- with LKE and CKE you 
don't have to choose, k8s executor is always there as an option for you.  So in 
this case seems like wouldn't be so crazy.
   
   But I think for me probably more than anything is, we don't want to just add 
`namespace` because it's not very future-proof, i.e. maybe there would be more 
info we'd want to add, and we don't want to add a column for it every time.  So 
probably there's a json column that wants to be there.  Maybe executor_data.  
Maybe we would move external_executor_id there.  And although it's maybe sort 
of tempting to expand executor_config to fill this role, probably makes sense 
to keep separate user config vs internal executor field.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] EugeneChung commented on issue #20468: Need to set `tags=[]` to remove all tags from a DAG.



EugeneChung commented on issue #20468:
URL: https://github.com/apache/airflow/issues/20468#issuecomment-1357051721

   You can remove tags from the `dag_tag` table.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #28452: TaskInstances do not succeed when using enable_logging=True option in DockerSwarmOperator



boring-cyborg[bot] commented on issue #28452:
URL: https://github.com/apache/airflow/issues/28452#issuecomment-1357049661

   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] chriscmorgan opened a new issue, #28452: TaskInstances do not succeed when using enable_logging=True option in DockerSwarmOperator



chriscmorgan opened a new issue, #28452:
URL: https://github.com/apache/airflow/issues/28452

   ### Apache Airflow Provider(s)
   
   docker
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-celery==3.1.0
   apache-airflow-providers-docker==3.3.0
   
   ### Apache Airflow version
   
   2.5.0
   
   ### Operating System
   
   centos 7
   
   ### Deployment
   
   Other Docker-based deployment
   
   ### Deployment details
   
   Running an a docker-swarm cluster deployed locally. 
   
   ### What happened
   
   Same issue as https://github.com/apache/airflow/issues/13675
   
   With logging_enabled=True the DAG never completes and stays in running. 
   
   When using DockerSwarmOperator together with the default enable_logging=True 
option, tasks do not succeed and stay in state running. When checking the 
docker service logs I can clearly see that the container ran and ended 
successfully. Airflow however does not recognize that the container finished 
and keeps the tasks in state running.
   
   ### What you think should happen instead
   
   DAG should complete.
   
   ### How to reproduce
   
   Docker-compose deployment:
   `
   curl -LfO 
'https://airflow.apache.org/docs/apache-airflow/2.5.0/docker-compose.yaml'
   docker compose up airflow-init
   docker compose up -d
   `
   
   DAG code:
   
   `
   from airflow import DAG
   from docker.types import Mount, SecretReference
   from airflow.providers.docker.operators.docker_swarm import 
DockerSwarmOperator
   from datetime import timedelta
   from airflow.utils.dates import days_ago
   from airflow.models import Variable
   from ssi_python_modules import task_fail_slack_alert
   # Get airflow variables
   etl_processes_version = Variable.get("etl_processes_version")
   etl_processes_ecr_repository = Variable.get("etl_processes_ecr_repository")
   
   # Setup default args for the job
   default_args = {
'owner': 'vcgstest',
'start_date': days_ago(2),
'retries': 0,
'on_failure_callback': task_fail_slack_alert
   }

   # Create the DAG
   dag = DAG(
   'patient_linking_dag', # DAG ID
   default_args=default_args,
   schedule_interval='0 0 * * *', # At midnight each day
   catchup=False
   )
   
   # # Create the DAG object
   with dag as dag:
  docker_swarm_task = DockerSwarmOperator(
task_id="job_run",

image=f'{etl_processes_ecr_repository}:{etl_processes_version}',
execution_timeout=timedelta(minutes=90),
command="",
api_version='auto',
tty=True,
enable_logging=True,
docker_url="tcp://:2376",
tls_ca_cert="/run/secrets/ca.pem",
tls_client_cert="/run/secrets/cert.pem",
tls_client_key="/run/secrets/key.pem"
)
   `
   
   
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] akakakakakaa opened a new pull request, #28451: Add `ensure_ascii=False` in trigger dag run API



akakakakakaa opened a new pull request, #28451:
URL: https://github.com/apache/airflow/pull/28451

   In Trigger DAG Run UI, non-unicode character broken. This change only affect 
to UI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Log FileTaskHandler to work with KubernetesExecutor's multi_namespace_mode (#28436)

2022-12-18 Thread xddeng

This is an automated email from the ASF dual-hosted git repository.

xddeng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 497c2c243d Log FileTaskHandler to work with KubernetesExecutor's 
multi_namespace_mode (#28436)
497c2c243d is described below

commit 497c2c243dd168639d34ff35e02e62d5177de338
Author: Xiaodong DENG 
AuthorDate: Mon Dec 19 04:29:09 2022 +0100

Log FileTaskHandler to work with KubernetesExecutor's multi_namespace_mode 
(#28436)
---
 airflow/utils/log/file_task_handler.py |  8 +++--
 tests/utils/test_log_handlers.py   | 61 +-
 2 files changed, 66 insertions(+), 3 deletions(-)

diff --git a/airflow/utils/log/file_task_handler.py 
b/airflow/utils/log/file_task_handler.py
index 6aee75ee33..b8feb2997b 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -191,16 +191,20 @@ class FileTaskHandler(logging.Handler):
 log += f"*** {str(e)}\n"
 return log, {"end_of_log": True}
 elif self._should_check_k8s(ti.queue):
+pod_override = ti.executor_config.get("pod_override")
+if pod_override and pod_override.metadata and 
pod_override.metadata.namespace:
+namespace = pod_override.metadata.namespace
+else:
+namespace = conf.get("kubernetes_executor", "namespace")
 try:
 from airflow.kubernetes.kube_client import get_kube_client
 
 kube_client = get_kube_client()
 
 log += f"*** Trying to get logs (last 100 lines) from worker 
pod {ti.hostname} ***\n\n"
-
 res = kube_client.read_namespaced_pod_log(
 name=ti.hostname,
-namespace=conf.get("kubernetes_executor", "namespace"),
+namespace=namespace,
 container="base",
 follow=False,
 tail_lines=100,
diff --git a/tests/utils/test_log_handlers.py b/tests/utils/test_log_handlers.py
index ee2ff2d9ce..8b7f0145de 100644
--- a/tests/utils/test_log_handlers.py
+++ b/tests/utils/test_log_handlers.py
@@ -21,9 +21,10 @@ import logging
 import logging.config
 import os
 import re
-from unittest.mock import patch
+from unittest.mock import MagicMock, patch
 
 import pytest
+from kubernetes.client import models as k8s
 
 from airflow.config_templates.airflow_local_settings import 
DEFAULT_LOGGING_CONFIG
 from airflow.models import DAG, DagRun, TaskInstance
@@ -219,6 +220,64 @@ class TestFileTaskLogHandler:
 # Remove the generated tmp log file.
 os.remove(log_filename)
 
+@pytest.mark.parametrize(
+"pod_override, namespace_to_call",
+[
+
pytest.param(k8s.V1Pod(metadata=k8s.V1ObjectMeta(namespace="namespace-A")), 
"namespace-A"),
+
pytest.param(k8s.V1Pod(metadata=k8s.V1ObjectMeta(namespace="namespace-B")), 
"namespace-B"),
+pytest.param(k8s.V1Pod(), "default"),
+pytest.param(None, "default"),
+
pytest.param(k8s.V1Pod(metadata=k8s.V1ObjectMeta(name="pod-name-xxx")), 
"default"),
+],
+)
+@patch.dict("os.environ", AIRFLOW__CORE__EXECUTOR="KubernetesExecutor")
+@patch("airflow.kubernetes.kube_client.get_kube_client")
+def test_read_from_k8s_under_multi_namespace_mode(
+self, mock_kube_client, pod_override, namespace_to_call
+):
+mock_read_namespaced_pod_log = MagicMock()
+mock_kube_client.return_value.read_namespaced_pod_log = 
mock_read_namespaced_pod_log
+
+def task_callable(ti):
+ti.log.info("test")
+
+dag = DAG("dag_for_testing_file_task_handler", start_date=DEFAULT_DATE)
+dagrun = dag.create_dagrun(
+run_type=DagRunType.MANUAL,
+state=State.RUNNING,
+execution_date=DEFAULT_DATE,
+)
+executor_config_pod = pod_override
+task = PythonOperator(
+task_id="task_for_testing_file_log_handler",
+dag=dag,
+python_callable=task_callable,
+executor_config={"pod_override": executor_config_pod},
+)
+ti = TaskInstance(task=task, run_id=dagrun.run_id)
+ti.try_number = 3
+
+logger = ti.log
+ti.log.disabled = False
+
+file_handler = next(
+(handler for handler in logger.handlers if handler.name == 
FILE_TASK_HANDLER), None
+)
+set_context(logger, ti)
+ti.run(ignore_ti_state=True)
+
+file_handler.read(ti, 3)
+
+# Check if kube_client.read_namespaced_pod_log() is called with the 
namespace we expect
+mock_read_namespaced_pod_log.assert_called_once_with(
+name=ti.hostname,
+namespace=namespace_to_call,
+container="base",
+follow=False,
+

[GitHub] [airflow] XD-DENG merged pull request #28436: Log FileTaskHandler to work with multi_namespace_mode when running using KubernetesExecutor



XD-DENG merged PR #28436:
URL: https://github.com/apache/airflow/pull/28436


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a diff in pull request #28416: Update index.rst



XD-DENG commented on code in PR #28416:
URL: https://github.com/apache/airflow/pull/28416#discussion_r1051753542


##
docs/helm-chart/index.rst:
##
@@ -122,10 +122,10 @@ The command removes all the Kubernetes components 
associated with the chart and
 .. note::
   Some kubernetes resources created by the chart `helm hooks 
`__
 might be left in the namespace after executing ``helm uninstall``, for 
example, ``brokerUrlSecret`` or ``fernetKeySecret``.
 
-Installing the Chart with Argo CD, Flux or Terraform
+Installing the Chart with Argo CD, Flux, Rancher or Terraform
 -

Review Comment:
   Thanks for the contribution.
   Please adjust the lengths of line 126 accordingly. Other the doc build may 
fail. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a diff in pull request #28436: Log FileTaskHandler to work with multi_namespace_mode when running using KubernetesExecutor



XD-DENG commented on code in PR #28436:
URL: https://github.com/apache/airflow/pull/28436#discussion_r1051751431


##
airflow/utils/log/file_task_handler.py:
##
@@ -191,16 +191,20 @@ def _read(self, ti: TaskInstance, try_number: int, 
metadata: dict[str, Any] | No
 log += f"*** {str(e)}\n"
 return log, {"end_of_log": True}
 elif self._should_check_k8s(ti.queue):
+pod_override = ti.executor_config.get("pod_override")
+if pod_override and pod_override.metadata and 
pod_override.metadata.namespace:
+namespace = pod_override.metadata.namespace
+else:
+namespace = conf.get("kubernetes_executor", "namespace")

Review Comment:
   Currently the user is specifying the namespace (if not default namespace in 
the config) in the executor config. When creating the TI, also storing the 
actual namespace bing used in the TI record sounds sensible to me.
   
   One concern is that this is only applicable when `KubernetesExecutor` is 
being used. Making it too "explicit" may confuse users who always only use 
other executors.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a diff in pull request #28436: Log FileTaskHandler to work with multi_namespace_mode when running using KubernetesExecutor



XD-DENG commented on code in PR #28436:
URL: https://github.com/apache/airflow/pull/28436#discussion_r1051751431


##
airflow/utils/log/file_task_handler.py:
##
@@ -191,16 +191,20 @@ def _read(self, ti: TaskInstance, try_number: int, 
metadata: dict[str, Any] | No
 log += f"*** {str(e)}\n"
 return log, {"end_of_log": True}
 elif self._should_check_k8s(ti.queue):
+pod_override = ti.executor_config.get("pod_override")
+if pod_override and pod_override.metadata and 
pod_override.metadata.namespace:
+namespace = pod_override.metadata.namespace
+else:
+namespace = conf.get("kubernetes_executor", "namespace")

Review Comment:
   Currently the user is specifying the namespace (if not default namespace in 
the config) in the executor config. When creating the TI, also storing the 
actual namespace bing used in the TI record sounds sensible to me.
   
   Only concern is that this is only applicable when `KubernetesExecutor` is 
being used. Making it too "explicit" may confuse users who always only use 
other executors.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28450: Remove extra H1 & improve formatting of Listeners docs page

boring-cyborg[bot] commented on PR #28450:
URL: https://github.com/apache/airflow/pull/28450#issuecomment-1357002489

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] nathan-contino opened a new pull request, #28450: Remove extra H1 & improve formatting of Listeners docs page

nathan-contino opened a new pull request, #28450:
URL: https://github.com/apache/airflow/pull/28450

I noticed that the documentation has an unclickable "Usage" page in the TOC.
A little digging later, I discovered that this page contains an extra H1, and
since this page is in the top level of the TOC, all the H1s on this page show
up in the left docs sidebar.

Demoted the "Usage" section to an H2, and fixed the other headers on this
page to use consistent underlining with most other docs pages in this repo. I
also took the liberty of sprucing up the language on the page to follow docs
best practices, like shorter, highly readable sentences, title case in section
titles, and bulleted lists to draw attention to important collections.

---
**^ Add meaningful description above**

Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch constraints-main updated: Updating constraints. Build id:

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new 9bc809b07e Updating constraints. Build id:
9bc809b07e is described below

commit 9bc809b07e21fa3863225cf291ac1e968b89865c
Author: Automated GitHub Actions commit 
AuthorDate: Mon Dec 19 02:02:21 2022 +

Updating constraints. Build id:

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
HEAD of '' in ''
with commit sha .

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.10.txt  | 4 ++--
 constraints-3.7.txt   | 2 +-
 constraints-3.8.txt   | 2 +-
 constraints-3.9.txt   | 2 +-
 constraints-no-providers-3.10.txt | 2 +-
 constraints-no-providers-3.7.txt  | 2 +-
 constraints-no-providers-3.8.txt  | 2 +-
 constraints-no-providers-3.9.txt  | 2 +-
 constraints-source-providers-3.10.txt | 4 ++--
 constraints-source-providers-3.7.txt  | 2 +-
 constraints-source-providers-3.8.txt  | 4 ++--
 constraints-source-providers-3.9.txt  | 4 ++--
 12 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/constraints-3.10.txt b/constraints-3.10.txt
index ffca69c2e2..55c380a306 100644
--- a/constraints-3.10.txt
+++ b/constraints-3.10.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T23:28:36Z
+# This constraints file was automatically generated on 2022-12-19T02:01:59Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -322,7 +322,7 @@ identify==2.5.10
 idna==3.4
 ijson==3.1.4
 imagesize==1.4.1
-importlib-metadata==5.1.0
+importlib-metadata==5.2.0
 incremental==22.10.0
 inflection==0.5.1
 influxdb-client==1.35.0
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index c4e0c7343a..8fde8e0f6c 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T23:28:56Z
+# This constraints file was automatically generated on 2022-12-19T02:02:19Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
diff --git a/constraints-3.8.txt b/constraints-3.8.txt
index c05395126a..3440a43687 100644
--- a/constraints-3.8.txt
+++ b/constraints-3.8.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T23:28:51Z
+# This constraints file was automatically generated on 2022-12-19T02:02:11Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
diff --git a/constraints-3.9.txt b/constraints-3.9.txt
index d69c8e8ac2..ce1facccb3 100644
--- a/constraints-3.9.txt
+++ b/constraints-3.9.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T23:28:48Z
+# This constraints file was automatically generated on 2022-12-19T02:02:10Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
diff --git a/constraints-no-providers-3.10.txt 
b/constraints-no-providers-3.10.txt
index deb0da2af5..0975deab9d 100644
--- a/constraints-no-providers-3.10.txt
+++ b/constraints-no-providers-3.10.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T23:26:39Z
+# This constraints file was automatically generated on 2022-12-19T02:00:04Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install just the 'bare' 'apache-airflow' package 
build from the HEAD of
 # the branch, without installing any of the providers.
diff --git a/constraints-no-providers-3.7.txt b/constraints-no-providers-3.7.txt
index 84af2110af..095c35a824 100644
--- a/constraints-no-providers-3.7.txt
+++ b/constraints-no-providers-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T23:27:07Z
+# This constraints file was automatically

[GitHub] [airflow] vchiapaikeo commented on issue #28393: Webserver reports "ImportError: Module "airflow.providers.google.cloud.operators.mlengine" does not define a "AIPlatformConsoleLink" attribute



vchiapaikeo commented on issue #28393:
URL: https://github.com/apache/airflow/issues/28393#issuecomment-1356961632

   Looks like the additions were properly added here: 
https://github.com/apache/airflow/pull/26836/files#diff-b8d2d329a113cf05d9bbd18b125843e86a3ce5c3c39704d29ad58bb83bb07861R1035-R1039
   
   PR for removal is here: https://github.com/apache/airflow/pull/28449


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vchiapaikeo opened a new pull request, #28449: Remove deprecated AIPlatformConsoleLinkk from google/provider.yaml



vchiapaikeo opened a new pull request, #28449:
URL: https://github.com/apache/airflow/pull/28449

   Exception is raised on webserver because a deprecated extra link was not 
removed from google/provider.yaml. This was removed as part of this PR: 
https://github.com/apache/airflow/pull/26836/files#diff-ee2dc8677bb15f1b09b5cf14d16fde2b48c0712cc3d4d2b08d478855634b4266L980
   
   Webserver Logs:
   ```
   10.253.8.251 - - [15/Dec/2022:18:32:58 +] "GET 
/object/next_run_datasets/recs_ranking_purchase_ranker_dag HTTP/1.1" 200 2 
"https://web.airflow.etsy-syseng-gke-prod.etsycloud.com/dags/recs_ranking_purchase_ranker_dag/code;
 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/108.0.0.0 Safari/537.36"
   raise ImportError(f'Module "{module_path}" does not define a 
"{class_name}" attribute/class')
 File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/module_loading.py",
 line 38, in import_string
   imported_class = import_string(class_name)
 File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/providers_manager.py",
 line 275, in _sanity_check
   Traceback (most recent call last):
   
   During handling of the above exception, another exception occurred:
   
   AttributeError: module 'airflow.providers.google.cloud.operators.mlengine' 
has no attribute 'AIPlatformConsoleLink'
   return getattr(module, class_name)
 File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/module_loading.py",
 line 36, in import_string
   Traceback (most recent call last):
   [2022-12-15 18:32:58,068] {providers_manager.py:243} WARNING - Exception 
when importing 
'airflow.providers.google.cloud.operators.mlengine.AIPlatformConsoleLink' from 
'apache-airflow-providers-google' package
   ImportError: Module "airflow.providers.google.cloud.operators.mlengine" does 
not define a "AIPlatformConsoleLink" attribute/class
   ```
   Fixes: https://github.com/apache/airflow/issues/28393
   
   cc @uranusjr
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'



uranusjr commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1356946147

   This looks like an issue in the Kubernetes package to me, from what I can 
tell there’s nothing out of the ordinary from Airflow’s perspective. I suspect 
the upgrade is not done correctly and the Kubernetes installation is broken.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Maintain manual scroll position in task logs (#28386)

2022-12-18 Thread bbovenzi

This is an automated email from the ASF dual-hosted git repository.

bbovenzi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 5c80d985a3 Maintain manual scroll position in task logs (#28386)
5c80d985a3 is described below

commit 5c80d985a3102a46f198aec1c57a255e00784c51
Author: Brent Bovenzi 
AuthorDate: Sun Dec 18 19:00:34 2022 -0600

Maintain manual scroll position in task logs (#28386)
---
 airflow/www/static/js/api/useTaskLog.ts| 20 -
 .../js/dag/details/taskInstance/Logs/LogBlock.tsx  | 90 ++
 .../js/dag/details/taskInstance/Logs/index.tsx | 45 +++
 3 files changed, 117 insertions(+), 38 deletions(-)

diff --git a/airflow/www/static/js/api/useTaskLog.ts 
b/airflow/www/static/js/api/useTaskLog.ts
index 580c5e0ab4..bbb6395878 100644
--- a/airflow/www/static/js/api/useTaskLog.ts
+++ b/airflow/www/static/js/api/useTaskLog.ts
@@ -17,6 +17,7 @@
  * under the License.
  */
 
+import { useState } from 'react';
 import axios, { AxiosResponse } from 'axios';
 import { useQuery } from 'react-query';
 import { useAutoRefresh } from 'src/context/autorefresh';
@@ -34,6 +35,7 @@ const useTaskLog = ({
   dagId, dagRunId, taskId, taskTryNumber, mapIndex, fullContent, state,
 }: Props) => {
   let url: string = '';
+  const [isPreviousStatePending, setPrevState] = useState(true);
   if (taskLogApi) {
 url = taskLogApi.replace('_DAG_RUN_ID_', dagRunId).replace('_TASK_ID_', 
taskId).replace(/-1$/, taskTryNumber.toString());
   }
@@ -49,12 +51,24 @@ const useTaskLog = ({
 || state === 'queued'
 || state === 'restarting';
 
+  // We also want to get the last log when the task was finished
+  const expectingLogs = isStatePending || isPreviousStatePending;
+
   return useQuery(
-['taskLogs', dagId, dagRunId, taskId, mapIndex, taskTryNumber, 
fullContent, state],
-() => axios.get(url, { headers: { Accept: 
'text/plain' }, params: { map_index: mapIndex, full_content: fullContent } }),
+['taskLogs', dagId, dagRunId, taskId, mapIndex, taskTryNumber, 
fullContent],
+() => {
+  setPrevState(isStatePending);
+  return axios.get(
+url,
+{
+  headers: { Accept: 'text/plain' },
+  params: { map_index: mapIndex, full_content: fullContent },
+},
+  );
+},
 {
   placeholderData: '',
-  refetchInterval: isStatePending && isRefreshOn && (autoRefreshInterval 
|| 1) * 1000,
+  refetchInterval: expectingLogs && isRefreshOn && (autoRefreshInterval || 
1) * 1000,
 },
   );
 };
diff --git a/airflow/www/static/js/dag/details/taskInstance/Logs/LogBlock.tsx 
b/airflow/www/static/js/dag/details/taskInstance/Logs/LogBlock.tsx
new file mode 100644
index 00..0ffa76e21f
--- /dev/null
+++ b/airflow/www/static/js/dag/details/taskInstance/Logs/LogBlock.tsx
@@ -0,0 +1,90 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import React, {
+  useRef, useEffect, useState,
+} from 'react';
+import {
+  Code,
+} from '@chakra-ui/react';
+
+import useOffsetHeight from 'src/utils/useOffsetHeight';
+
+interface Props {
+  parsedLogs: string;
+  wrap: boolean;
+  tryNumber: number;
+}
+
+const LogBlock = ({
+  parsedLogs,
+  wrap,
+  tryNumber,
+}: Props) => {
+  const [autoScroll, setAutoScroll] = useState(true);
+  const logBoxRef = useRef(null);
+
+  const maxHeight = useOffsetHeight(logBoxRef, parsedLogs);
+
+  const codeBlockBottomDiv = useRef(null);
+
+  const scrollToBottom = () => {
+codeBlockBottomDiv.current?.scrollIntoView({ block: 'nearest', inline: 
'nearest' });
+  };
+
+  useEffect(() => {
+// Always scroll to bottom when wrap or tryNumber change
+scrollToBottom();
+  }, [wrap, tryNumber]);
+
+  useEffect(() => {
+// When logs change, only scroll if autoScroll is enabled
+if (autoScroll) scrollToBottom();
+  }, [parsedLogs, autoScroll]);
+
+  const onScroll = (e: React.UIEvent) => {
+if (e.currentTarget) {
+  const { scrollTop, offsetHeight, scrollHeight } = e.currentTarget;
+  // Enable autoscroll if we've scrolled to the bottom of the logs
+  setAutoScroll(scrollTop

[GitHub] [airflow] bbovenzi closed issue #28143: Logs tab is automatically scrolling to the bottom while user is reading logs



bbovenzi closed issue #28143: Logs tab is automatically scrolling to the bottom 
while user is reading logs
URL: https://github.com/apache/airflow/issues/28143


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bbovenzi merged pull request #28386: Maintain manual scroll position in task logs



bbovenzi merged PR #28386:
URL: https://github.com/apache/airflow/pull/28386


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #28393: Webserver reports "ImportError: Module "airflow.providers.google.cloud.operators.mlengine" does not define a "AIPlatformConsoleLink" attribute/cl



uranusjr commented on issue #28393:
URL: https://github.com/apache/airflow/issues/28393#issuecomment-1356935480

   You can submit a PR to do the removal (and additions—see #26836 for what 
hooks were added) and the pre-commit check can be dealt separately later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #12644: Network instabilities are able to freeze KubernetesJobWatcher



github-actions[bot] commented on issue #12644:
URL: https://github.com/apache/airflow/issues/12644#issuecomment-1356911790

   This issue has been automatically marked as stale because it has been open 
for 30 days with no response from the author. It will be closed in next 7 days 
if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #13818: Default user is not working with AUTH_LDAP - Airflow 2.0



github-actions[bot] commented on issue #13818:
URL: https://github.com/apache/airflow/issues/13818#issuecomment-1356911780

   This issue has been automatically marked as stale because it has been open 
for 30 days with no response from the author. It will be closed in next 7 days 
if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28182: Better error message for bad stat name



potiuk commented on PR #28182:
URL: https://github.com/apache/airflow/pull/28182#issuecomment-1356907529

   And just to explain a bit - I realised closing the issue could have sounded 
a bit rude. But it was not really and in my mind it was pretty clear at that 
time (and this  is how I interpreted the title of the PR and the "oops" + 
opening another issue by TP.  But I really thought the excessive code - 
especially the `getattr` to provide backwards compatibility was totally not 
worth it. Effectively the trade-off  was:
   
   a)const definition stays
   
   vs.
   
   b) importing regexp library + adding getattr method in which the same 
constant was effectively repeated,  adding new method where not only logic for 
gettattr + adding tests if regexp works  + message that required the user to 
parse the regexp
   
   The title of the PR was "better message" which was not too much better 
(telling regexp in a user message is a sign that we expect the user can parse 
the regexp, which is kinda non-human.
   
   Simply speaking the change from TP looked so much more doing what your PR 
title suggested - "WAY better message".
   
   There could be other reasons why we could get rid of the constant, but I 
think just "better user message" is not one of those reasons.
   
   Sorry if you took that as being rude or smth like that. It was just 
perfectly logical choice for me. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch constraints-main updated: Updating constraints. Build id:

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new b620792452 Updating constraints. Build id:
b620792452 is described below

commit b6207924529123ed1a97b66a9fbe0f6d7c54213b
Author: Automated GitHub Actions commit 
AuthorDate: Sun Dec 18 23:28:58 2022 +

Updating constraints. Build id:

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
HEAD of '' in ''
with commit sha .

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.10.txt  | 10 +-
 constraints-3.7.txt   |  8 
 constraints-3.8.txt   | 12 ++--
 constraints-3.9.txt   | 12 ++--
 constraints-no-providers-3.10.txt |  4 ++--
 constraints-no-providers-3.7.txt  |  4 ++--
 constraints-no-providers-3.8.txt  |  6 +++---
 constraints-no-providers-3.9.txt  |  6 +++---
 constraints-source-providers-3.10.txt | 10 +-
 constraints-source-providers-3.7.txt  | 10 +-
 constraints-source-providers-3.8.txt  | 10 +-
 constraints-source-providers-3.9.txt  | 10 +-
 12 files changed, 51 insertions(+), 51 deletions(-)

diff --git a/constraints-3.10.txt b/constraints-3.10.txt
index 268411165a..ffca69c2e2 100644
--- a/constraints-3.10.txt
+++ b/constraints-3.10.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T21:48:06Z
+# This constraints file was automatically generated on 2022-12-18T23:28:36Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -203,7 +203,7 @@ colorama==0.4.6
 colorlog==4.8.0
 commonmark==0.9.1
 connexion==2.14.1
-coverage==6.5.0
+coverage==7.0.0
 crcmod==1.7
 cron-descriptor==1.2.32
 croniter==1.3.8
@@ -348,7 +348,7 @@ jsonschema==4.17.3
 junit-xml==1.9
 jupyter-client==7.3.4
 jupyter_core==5.1.0
-keyring==23.11.0
+keyring==23.13.1
 kombu==5.2.4
 krb5==0.4.1
 kubernetes==23.6.0
@@ -469,7 +469,7 @@ pyparsing==3.0.9
 pypsrp==0.8.1
 pyrsistent==0.19.2
 pyspark==3.3.1
-pyspnego==0.6.3
+pyspnego==0.7.0
 pytest-asyncio==0.20.3
 pytest-capture-warnings==0.0.4
 pytest-cov==4.0.0
@@ -591,7 +591,7 @@ types-cryptography==3.3.23.2
 types-docutils==0.19.1.1
 types-paramiko==2.12.0.1
 types-protobuf==4.21.0.2
-types-python-dateutil==2.8.19.4
+types-python-dateutil==2.8.19.5
 types-python-slugify==7.0.0.1
 types-pytz==2022.6.0.1
 types-redis==4.3.21.6
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index f57367a02b..c4e0c7343a 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T21:48:41Z
+# This constraints file was automatically generated on 2022-12-18T23:28:56Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -204,7 +204,7 @@ colorama==0.4.6
 colorlog==4.8.0
 commonmark==0.9.1
 connexion==2.14.1
-coverage==6.5.0
+coverage==7.0.0
 crcmod==1.7
 cron-descriptor==1.2.32
 croniter==1.3.8
@@ -470,7 +470,7 @@ pyparsing==3.0.9
 pypsrp==0.8.1
 pyrsistent==0.19.2
 pyspark==3.3.1
-pyspnego==0.6.3
+pyspnego==0.7.0
 pytest-asyncio==0.20.3
 pytest-capture-warnings==0.0.4
 pytest-cov==4.0.0
@@ -592,7 +592,7 @@ types-cryptography==3.3.23.2
 types-docutils==0.19.1.1
 types-paramiko==2.12.0.1
 types-protobuf==4.21.0.2
-types-python-dateutil==2.8.19.4
+types-python-dateutil==2.8.19.5
 types-python-slugify==7.0.0.1
 types-pytz==2022.6.0.1
 types-redis==4.3.21.6
diff --git a/constraints-3.8.txt b/constraints-3.8.txt
index aa1e79b4f3..c05395126a 100644
--- a/constraints-3.8.txt
+++ b/constraints-3.8.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T21:48:33Z
+# This constraints file was automatically generated on 2022-12-18T23:28:51Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -204,7 +204,7 @@ colorama==0.4.6
 colorlog==4.8.0
 commonmark==0.9.1
 connexion==2.14.1
-coverage==6.5.0
+coverage==7.0.0
 crcmod==1.7
 cron-descriptor==1.2.32
 croniter==1.3.8

[GitHub] [airflow] potiuk commented on pull request #28283: Trigger gevent monkeypatching via environment variable



potiuk commented on PR #28283:
URL: https://github.com/apache/airflow/pull/28283#issuecomment-1356898026

   Seems we have confirmation that at least this part of the gevent 
configuration (monkeypatching) works with this approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #28318: Add FTPSFileTransmitOperator



RachitSharma2001 commented on code in PR #28318:
URL: https://github.com/apache/airflow/pull/28318#discussion_r1051680027


##
tests/providers/ftp/operators/test_ftp.py:
##
@@ -184,3 +180,108 @@ def test_unequal_local_remote_file_paths(self):
 local_filepath=["/tmp/test1", "/tmp/test2"],
 remote_filepath="/tmp/test1",
 )
+
+
+class TestFTPSFileTransmitOperator:
+def setup_method(self):
+self.test_local_dir = "/tmp"
+self.test_local_dir_int = "/tmp/interdir"
+self.test_remote_dir = "/ftpshome"
+self.test_remote_dir_int = "/ftpshome/interdir"
+self.test_local_filename = "test_local_file"
+self.test_remote_filename = "test_remote_file"
+self.test_local_filepath = 
f"{self.test_local_dir}/{self.test_local_filename}"
+self.test_remote_filepath = 
f"{self.test_remote_dir}/{self.test_remote_filename}"
+self.test_local_filepath_int_dir = 
f"{self.test_local_dir_int}/{self.test_local_filename}"
+self.test_remote_filepath_int_dir = 
f"{self.test_remote_dir_int}/{self.test_remote_filename}"
+
+def teardown_method(self):
+if os.path.exists(self.test_local_dir_int):
+os.rmdir(self.test_local_dir_int)

Review Comment:
   @Taragolis Thank you for letting me know, that is a good point. I have 
pushed a change where any directories that are created are created within 
`tmp_path`. You can see the change in 
`test_file_transfer_with_intermediate_dir_get`. Let me know if there is 
anything else needed to be added.



##
tests/providers/ftp/operators/test_ftp.py:
##
@@ -184,3 +180,108 @@ def test_unequal_local_remote_file_paths(self):
 local_filepath=["/tmp/test1", "/tmp/test2"],
 remote_filepath="/tmp/test1",
 )
+
+
+class TestFTPSFileTransmitOperator:
+def setup_method(self):
+self.test_local_dir = "/tmp"
+self.test_local_dir_int = "/tmp/interdir"
+self.test_remote_dir = "/ftpshome"
+self.test_remote_dir_int = "/ftpshome/interdir"
+self.test_local_filename = "test_local_file"
+self.test_remote_filename = "test_remote_file"
+self.test_local_filepath = 
f"{self.test_local_dir}/{self.test_local_filename}"
+self.test_remote_filepath = 
f"{self.test_remote_dir}/{self.test_remote_filename}"
+self.test_local_filepath_int_dir = 
f"{self.test_local_dir_int}/{self.test_local_filename}"
+self.test_remote_filepath_int_dir = 
f"{self.test_remote_dir_int}/{self.test_remote_filename}"
+
+def teardown_method(self):
+if os.path.exists(self.test_local_dir_int):
+os.rmdir(self.test_local_dir_int)

Review Comment:
   @Taragolis Thank you for letting me know, that is a good point. I have 
pushed a change where any directories that are created are done so within 
`tmp_path`. You can see the change in 
`test_file_transfer_with_intermediate_dir_get`. Let me know if there is 
anything else needed to be added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] RachitSharma2001 commented on a diff in pull request #28318: Add FTPSFileTransmitOperator



RachitSharma2001 commented on code in PR #28318:
URL: https://github.com/apache/airflow/pull/28318#discussion_r1051680027


##
tests/providers/ftp/operators/test_ftp.py:
##
@@ -184,3 +180,108 @@ def test_unequal_local_remote_file_paths(self):
 local_filepath=["/tmp/test1", "/tmp/test2"],
 remote_filepath="/tmp/test1",
 )
+
+
+class TestFTPSFileTransmitOperator:
+def setup_method(self):
+self.test_local_dir = "/tmp"
+self.test_local_dir_int = "/tmp/interdir"
+self.test_remote_dir = "/ftpshome"
+self.test_remote_dir_int = "/ftpshome/interdir"
+self.test_local_filename = "test_local_file"
+self.test_remote_filename = "test_remote_file"
+self.test_local_filepath = 
f"{self.test_local_dir}/{self.test_local_filename}"
+self.test_remote_filepath = 
f"{self.test_remote_dir}/{self.test_remote_filename}"
+self.test_local_filepath_int_dir = 
f"{self.test_local_dir_int}/{self.test_local_filename}"
+self.test_remote_filepath_int_dir = 
f"{self.test_remote_dir_int}/{self.test_remote_filename}"
+
+def teardown_method(self):
+if os.path.exists(self.test_local_dir_int):
+os.rmdir(self.test_local_dir_int)

Review Comment:
   @Taragolis Thank you for letting me know, that is a good point. I have 
pushed a change where `tmp_path `is used and any directories that are created 
are created within this `tmp_path`. You can see the change in 
`test_file_transfer_with_intermediate_dir_get`. Let me know if there is 
anything else needed to be added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] victorjourne commented on pull request #28283: Trigger gevent monkeypatching via environment variable



victorjourne commented on PR #28283:
URL: https://github.com/apache/airflow/pull/28283#issuecomment-1356892685

   I can confirm that this fix solves my issue #28380 and #8164 warnings have 
disappeared.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] victorjourne commented on issue #28380: Celery green threads incompatibility



victorjourne commented on issue #28380:
URL: https://github.com/apache/airflow/issues/28380#issuecomment-1356891596

   After testing many configurations about celery backend, the solution I found 
is the combination of :
   - Force celery **not to store the task return** by passing the env variable 
: `CELERY_IGNORE_RESULT='True'` 
([docs](https://docs.celeryq.dev/en/stable/userguide/configuration.html#std-setting-task_ignore_result)).
 
   Fortunately, airflow stores through Xcom the task result to process it 
further.
   - Use this bug fix from @potiuk  :  #28283
   
   However, it seems to be inconsistent with this airflow 
[diagram](https://airflow.apache.org/docs/apache-airflow/stable/executor/celery.html#task-execution-process),
 the task status in the metadata should be updated from the result backend 
table, which is empty I would go deeper in the code if I had the time.
   
   In any case, the whole issue about celery green threads is related to the 
way of airflow calls the **result backend**. There is something blocking the 
celery workers to stop. I should to investigate it more, but quite astonish to 
be the first user to undergo this, since it is a quite common pattern to 
concurrently call **IO tasks** with green threads. To achieve that, you guys 
may use `CeleryExecutor`, or the `LocalExecutor`?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed issue #28232: Allow changing the Task comments via a task



potiuk closed issue #28232: Allow changing the Task comments via a task
URL: https://github.com/apache/airflow/issues/28232


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #28232: Allow changing the Task comments via a task



potiuk commented on issue #28232:
URL: https://github.com/apache/airflow/issues/28232#issuecomment-1356890786

   IMHO - This should be done via API not via decorator/task. There is no 
"internal" mechanism to change some tasks from other tasks and stable REST API 
is the best way of doing it. Let me convert it to a discussion - maybe others 
will see it differently but I have a feeling that the way it is specified (what 
should be decorateD? when? when it should be processed? etc. is far from 
"defined". This is more a wish than defined feature, and discussion is more 
appropriate place for it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (21f0600f69 -> 5b54e8d21b)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 21f0600f69 Correctly template Glue Jobs `create_job_kwargs` arg 
(#28403)
 add 5b54e8d21b fix: very small typo error in readme (#28447)

No new revisions were added by this update.

Summary of changes:
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28447: fix: very small typo error in readme



boring-cyborg[bot] commented on PR #28447:
URL: https://github.com/apache/airflow/pull/28447#issuecomment-1356889896

   Awesome work, congrats on your first merged pull request!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #28447: fix: very small typo error in readme



potiuk merged PR #28447:
URL: https://github.com/apache/airflow/pull/28447


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] 0xflotus opened a new pull request, #28447: fix: very small typo error in readme



0xflotus opened a new pull request, #28447:
URL: https://github.com/apache/airflow/pull/28447

   I only fixed a small typo errror in the readme file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28447: fix: very small typo error in readme

boring-cyborg[bot] commented on PR #28447:
URL: https://github.com/apache/airflow/pull/28447#issuecomment-1356888244

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #26993: Fix ti._try_number for deferred and up_for_reschedule tasks



potiuk commented on PR #26993:
URL: https://github.com/apache/airflow/pull/26993#issuecomment-1356887634

   some errors @bbovenzi :( 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'



potiuk commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1356887060

   Also @KulykDmytro - could you please explain the exact deplouyment you have 
- which versionsm, what kind of chart or any other deployment you have, how 
exactly did you upgrade (did you use constraints?). For me this might also be 
caused by mis-configuration of the k8s version you have and k8s client 
libraries you have. It almost look like you have an old client library while 
you are using newer k8S. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'



potiuk commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1356886524

   Ah I see you seeem to do run it. @dstandish @jedcunningham - does it ring a 
bell ? 
   
   Did you have anything specfic in your upgrade also around the k8s version 
change during migration @KulykDmytro ? Ephemeral containers seems to be a new 
feature of K8S so maybe this somehow impacts the problem.
   
   Did you also run "reserialize" after or before the migration?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'



potiuk commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1356885469

   Did you try `airflow dags reserialize` command @KulykDmytro after the 
upgrade? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch constraints-main updated: Updating constraints. Build id:

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new 913fdd3401 Updating constraints. Build id:
913fdd3401 is described below

commit 913fdd3401ec2e9067fed545da400824d3fd8bbb
Author: Automated GitHub Actions commit 
AuthorDate: Sun Dec 18 21:48:45 2022 +

Updating constraints. Build id:

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
HEAD of '' in ''
with commit sha .

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.10.txt  | 2 +-
 constraints-3.7.txt   | 4 ++--
 constraints-3.8.txt   | 2 +-
 constraints-3.9.txt   | 2 +-
 constraints-no-providers-3.10.txt | 2 +-
 constraints-no-providers-3.7.txt  | 2 +-
 constraints-no-providers-3.8.txt  | 2 +-
 constraints-no-providers-3.9.txt  | 2 +-
 constraints-source-providers-3.10.txt | 2 +-
 constraints-source-providers-3.7.txt  | 2 +-
 constraints-source-providers-3.8.txt  | 2 +-
 constraints-source-providers-3.9.txt  | 2 +-
 12 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/constraints-3.10.txt b/constraints-3.10.txt
index 32572cba51..268411165a 100644
--- a/constraints-3.10.txt
+++ b/constraints-3.10.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T18:52:56Z
+# This constraints file was automatically generated on 2022-12-18T21:48:06Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index 48e0eb3dc7..f57367a02b 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T18:53:36Z
+# This constraints file was automatically generated on 2022-12-18T21:48:41Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -349,7 +349,7 @@ jsonschema==4.17.3
 junit-xml==1.9
 jupyter-client==7.3.4
 jupyter_core==4.12.0
-keyring==23.12.1
+keyring==23.13.1
 kombu==5.2.4
 krb5==0.4.1
 kubernetes==23.6.0
diff --git a/constraints-3.8.txt b/constraints-3.8.txt
index 3ecb7195a4..aa1e79b4f3 100644
--- a/constraints-3.8.txt
+++ b/constraints-3.8.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T18:53:28Z
+# This constraints file was automatically generated on 2022-12-18T21:48:33Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
diff --git a/constraints-3.9.txt b/constraints-3.9.txt
index d1e05d67ea..81d61d0e5e 100644
--- a/constraints-3.9.txt
+++ b/constraints-3.9.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T18:53:25Z
+# This constraints file was automatically generated on 2022-12-18T21:48:28Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
diff --git a/constraints-no-providers-3.10.txt 
b/constraints-no-providers-3.10.txt
index 33d95fb130..593cac17a4 100644
--- a/constraints-no-providers-3.10.txt
+++ b/constraints-no-providers-3.10.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T18:49:58Z
+# This constraints file was automatically generated on 2022-12-18T21:45:11Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install just the 'bare' 'apache-airflow' package 
build from the HEAD of
 # the branch, without installing any of the providers.
diff --git a/constraints-no-providers-3.7.txt b/constraints-no-providers-3.7.txt
index 4cdee8efdf..88ffd6105f 100644
--- a/constraints-no-providers-3.7.txt
+++ b/constraints-no-providers-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-18T18:50:50Z
+# This constraints file was automatically generated on

[GitHub] [airflow] potiuk closed issue #28201: Tasks get terminated with SIGTERM



potiuk closed issue #28201: Tasks get terminated with SIGTERM
URL: https://github.com/apache/airflow/issues/28201


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #28201: Tasks get terminated with SIGTERM



potiuk commented on issue #28201:
URL: https://github.com/apache/airflow/issues/28201#issuecomment-1356881079

   Converting into discussion as it seems it is just misconfiguration. We can 
always re-open if we have evidences it's not.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #28201: Tasks get terminated with SIGTERM



potiuk commented on issue #28201:
URL: https://github.com/apache/airflow/issues/28201#issuecomment-1356880839

   Looks like the problem is connected somehow with bad configuration of your 
S3 logs. It seems to fail when logging fails with 403 - forbidden error. Fix 
the problem (by configuring the credentials with the right access - look up the 
logging documentation for details) and see if it solves the problem. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28182: Better error message for bad stat name



potiuk commented on PR #28182:
URL: https://github.com/apache/airflow/pull/28182#issuecomment-1356876053

   > you don't think that, given that the work has already been done, it's more 
positive than negative to remove this constant, which ultimately is not really 
needed and is somewhat vague?
   
   Sorry missed that one as I did not realise it was for me, but I am catching 
up now after moving to a different country :).
   
   I just think it's a lot of code to add for such a simple thing. The way it 
is defined now explains it in the code pretty well (for maintainers) and the 
error message is very clear about it (for users). It serves the purpose pretty 
well. This is one line of code change rather than 40 lines of code added 10 
removed here (net + 30 lines)
   
   That was my primary thinking by choosing "better". I usually do not think 
that  something is "done" is a reason for its own to classify it as "better". 
Quite often my own "done code" is reworked, simplified and removed - sometime 
during the same PR, sometimes months or years later. And I am completely 
indifferent to the fact that it's "done". Because "done" is usually few percent 
of the effort needed for lifecycle of the code. I believe code is usually a 
liability  that has to be maintained rather than asset. If it can be done in 
far less code - why not remove something that is done.
   
   If something serves the same purpose and is clear with far less code or no 
code at al - then for me it is "better". 
   
   I actually love to remove my own code when possible while achieving same 
result. For me that means one less thing I have to maintain in the future (and 
that extrapolates to all the maintainers of Airflow).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed issue #28059: Massive slowdown of DAG File processor due to JSON schema upgrade in Airflow's core.



potiuk closed issue #28059: Massive slowdown of DAG File processor due to JSON 
schema upgrade in Airflow's core.
URL: https://github.com/apache/airflow/issues/28059


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #28059: Massive slowdown of DAG File processor due to JSON schema upgrade in Airflow's core.



potiuk commented on issue #28059:
URL: https://github.com/apache/airflow/issues/28059#issuecomment-1356873332

   Since we have not heard from the user - converting it into a discussion. 
Shall there be more information/data provided we can consider what to do with 
it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (bea91b9018 -> 21f0600f69)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from bea91b9018 Re-enable Plyvel on ARM as it now builds cleanly (#28443)
 add 21f0600f69 Correctly template Glue Jobs `create_job_kwargs` arg 
(#28403)

No new revisions were added by this update.

Summary of changes:
 airflow/providers/amazon/aws/operators/glue.py|  1 +
 tests/providers/amazon/aws/operators/test_glue.py | 48 +--
 2 files changed, 38 insertions(+), 11 deletions(-)

[GitHub] [airflow] potiuk merged pull request #28403: Correctly template Glue Jobs `create_job_kwargs` arg



potiuk merged PR #28403:
URL: https://github.com/apache/airflow/pull/28403


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (2a78f50b36 -> bea91b9018)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from 2a78f50b36 Fix discoverability of tests for ARM in Breeze (#28432)
 add bea91b9018 Re-enable Plyvel on ARM as it now builds cleanly (#28443)

No new revisions were added by this update.

Summary of changes:
 Dockerfile| 2 +-
 Dockerfile.ci | 2 +-
 scripts/docker/install_os_dependencies.sh | 2 +-
 setup.py  | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

[GitHub] [airflow] potiuk merged pull request #28443: Re-enable Plyvel on ARM as it now builds cleanly



potiuk merged PR #28443:
URL: https://github.com/apache/airflow/pull/28443


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28034: Added custom environment variables to statsd



potiuk commented on PR #28034:
URL: https://github.com/apache/airflow/pull/28034#issuecomment-1356864111

   Are you still working on it @RagingPuppies  ? If so - it needs 
rebase/conflict resolving


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28442: Re-enable azure service bus on ARM as it now builds cleanly



potiuk commented on PR #28442:
URL: https://github.com/apache/airflow/pull/28442#issuecomment-1356858609

   > Oh, I just repeat step which probably most of user do first, just try to 
install it by `pip` into the regular image.
   
   I see what you mean. But the same error might happen for a number of other 
packages - not even our providers. There are plenty of packages there that 
require being compiled (unfortunately). And even some of our providers do. And 
some of them require some system packages to be installed. I do not think we 
can reasonably describe it for each provider separately - especially that many 
of our users do not use the image at all - they install airflow in their own 
virtualenv and on different distributions and there different dependencies are 
needed for those different distributions.
   
   I am afraid (but maybe I am exaggerating) that if we start describing what 
else is needed, we might make people think that we should describe all the 
prerequisites for all the possible distributions, and that this description is 
"complete" - i.e. explains all the necessary requirements. For example you 
won't be able to install mysql provider without having mysql client libraries 
installed - and there are different ways how those can be installed on centos, 
redhat, debian, mint, etc. not even mentioning MacOS installation - which 
brings the whole new host of problems (additionally M1 + Intel).
   
   I think if we start describing things like that in the docs, we are going 
down the rabbit hole. Should we verify and check all the prerequisites and 
describe them for all providers? Doing it for one provider will - pretty 
inevitably - lead to people asking:
   
   > My provider fails to install and I see those other providers have 
instructions - surely you should have instruction here and I should not need to 
solve it myself when I am installing it on . 
- what are the instructions?"
   
   I think it's a dangerous path to walk.
   
   BuI just thought about another idea.
   
   Something that I have thought about before. Maybe - on top of having 
optimized apache/airflow image and "slim" image we should have "fat" image 
(maybe a bit better named) that will have build-essentials installed and all of 
the librarires CI image has installed in "dev". Then the solution for anyone 
who wants to install such package will be - "customize" (you will have 
optimized image), "extend with build essentials" (less optimized) and "use fat 
image" (huge image, but will install any provider by default). 
   
   Then we can even add a generic FAQ on "build error" when adding your package 
(or maybe even we could try to invent some smart way of capturing the error in 
the PROD image and displaying the instruction to the user. I **think** it could 
be done by changing shelll in the image to our custom command. 
   
   I think building such "generic" message what to do in case of build error 
and three options to follow - one of them as easy as changing `FROM 
apache/airflow:2.5.0` to `FROM apache/airflow:fat-2.5.0` - should do the job 
nicely.
   
   WDYT? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] IAL32 commented on a diff in pull request #28403: Correctly template Glue Jobs `create_job_kwargs` arg



IAL32 commented on code in PR #28403:
URL: https://github.com/apache/airflow/pull/28403#discussion_r1051651346


##
tests/providers/amazon/aws/operators/test_glue.py:
##
@@ -16,20 +16,62 @@
 # under the License.
 from __future__ import annotations
 
+import copy
 from unittest import mock
 
 import pytest
 
 from airflow.configuration import conf
+from airflow.models import DAG, DagRun, TaskInstance
 from airflow.providers.amazon.aws.hooks.glue import GlueJobHook
 from airflow.providers.amazon.aws.hooks.s3 import S3Hook
 from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
+from airflow.utils import timezone
+
+TASK_ID = "test_glue_operator"
+DAG_ID = "test_dag_id"
+JOB_NAME = "test_job_name"
+DEFAULT_DATE = timezone.datetime(2017, 1, 1)
+
+BASE_LOCATION = "some-bucket/{{ task_instance_key_str }}"
+SCRIPT_LOCATION = f"s3://{BASE_LOCATION}/script-location.py"
+CREATE_JOB_KWARGS = {
+"GlueVersion": "4.0",
+"Command": {
+"Name": "glueetl",
+"ScriptLocation": SCRIPT_LOCATION,
+},
+"WorkerType": "G.1X",
+"NumberOfWorkers": 2,
+}
+EXPECTED_BASE_LOCATION = f"some-bucket/{DAG_ID}__{TASK_ID}__20170101"
 
 
 class TestGlueJobOperator:
 def setup_method(self):
 conf.load_test_config()
 
+def test_render_template(self):
+args = {"owner": "airflow", "start_date": DEFAULT_DATE}
+
+dag = DAG(DAG_ID, default_args=args)
+mock_operator = GlueJobOperator(
+task_id=TASK_ID, dag=dag, create_job_kwargs=CREATE_JOB_KWARGS, 
s3_bucket=BASE_LOCATION
+)
+
+dag_run = DagRun(dag_id=mock_operator.dag_id, 
execution_date=DEFAULT_DATE, run_id="test")
+ti = TaskInstance(task=mock_operator)
+ti.dag_run = dag_run

Review Comment:
   Done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch constraints-main updated: Updating constraints. Build id:

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new 06a0afc94c Updating constraints. Build id:
06a0afc94c is described below

commit 06a0afc94c10781f245c72d9248f27054a4e5cca
Author: Automated GitHub Actions commit 
AuthorDate: Sun Dec 18 18:53:39 2022 +

Updating constraints. Build id:

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
HEAD of '' in ''
with commit sha .

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.10.txt  |  8 
 constraints-3.7.txt   | 10 +-
 constraints-3.8.txt   |  8 
 constraints-3.9.txt   |  8 
 constraints-no-providers-3.10.txt |  6 +++---
 constraints-no-providers-3.7.txt  |  4 ++--
 constraints-no-providers-3.8.txt  |  6 +++---
 constraints-no-providers-3.9.txt  |  6 +++---
 constraints-source-providers-3.10.txt |  8 
 constraints-source-providers-3.7.txt  |  8 
 constraints-source-providers-3.8.txt  |  8 
 constraints-source-providers-3.9.txt  |  8 
 12 files changed, 44 insertions(+), 44 deletions(-)

diff --git a/constraints-3.10.txt b/constraints-3.10.txt
index feb30adce2..32572cba51 100644
--- a/constraints-3.10.txt
+++ b/constraints-3.10.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-16T23:36:18Z
+# This constraints file was automatically generated on 2022-12-18T18:52:56Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -170,7 +170,7 @@ backoff==1.10.0
 bcrypt==4.0.1
 beautifulsoup4==4.11.1
 billiard==3.6.4.0
-black==22.12.0
+black==23.1a1
 bleach==5.0.1
 blinker==1.5
 boto3==1.26.32
@@ -374,7 +374,7 @@ mongomock==4.1.2
 monotonic==1.6
 more-itertools==8.14.0
 moreorless==0.4.0
-moto==4.0.11
+moto==4.0.12
 msal-extensions==1.0.0
 msal==1.20.0
 msgpack==1.0.4
@@ -491,7 +491,7 @@ python-nvd3==0.15.0
 python-slugify==7.0.0
 python-telegram-bot==13.15
 pytz-deprecation-shim==0.1.0.post0
-pytz==2022.6
+pytz==2022.7
 pytzdata==2020.1
 pywinrm==0.4.3
 pyzmq==24.0.1
diff --git a/constraints-3.7.txt b/constraints-3.7.txt
index bf43ed299d..48e0eb3dc7 100644
--- a/constraints-3.7.txt
+++ b/constraints-3.7.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-16T23:36:56Z
+# This constraints file was automatically generated on 2022-12-18T18:53:36Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -170,7 +170,7 @@ backports.zoneinfo==0.2.1
 bcrypt==4.0.1
 beautifulsoup4==4.11.1
 billiard==3.6.4.0
-black==22.12.0
+black==23.1a1
 bleach==5.0.1
 blinker==1.5
 boto3==1.26.32
@@ -349,7 +349,7 @@ jsonschema==4.17.3
 junit-xml==1.9
 jupyter-client==7.3.4
 jupyter_core==4.12.0
-keyring==23.11.0
+keyring==23.12.1
 kombu==5.2.4
 krb5==0.4.1
 kubernetes==23.6.0
@@ -375,7 +375,7 @@ mongomock==4.1.2
 monotonic==1.6
 more-itertools==9.0.0
 moreorless==0.4.0
-moto==4.0.11
+moto==4.0.12
 msal-extensions==1.0.0
 msal==1.20.0
 msgpack==1.0.4
@@ -492,7 +492,7 @@ python-nvd3==0.15.0
 python-slugify==7.0.0
 python-telegram-bot==13.15
 pytz-deprecation-shim==0.1.0.post0
-pytz==2022.6
+pytz==2022.7
 pytzdata==2020.1
 pywinrm==0.4.3
 pyzmq==24.0.1
diff --git a/constraints-3.8.txt b/constraints-3.8.txt
index 63587341db..3ecb7195a4 100644
--- a/constraints-3.8.txt
+++ b/constraints-3.8.txt
@@ -1,5 +1,5 @@
 #
-# This constraints file was automatically generated on 2022-12-16T23:36:49Z
+# This constraints file was automatically generated on 2022-12-18T18:53:28Z
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -171,7 +171,7 @@ backports.zoneinfo==0.2.1
 bcrypt==4.0.1
 beautifulsoup4==4.11.1
 billiard==3.6.4.0
-black==22.12.0
+black==23.1a1
 bleach==5.0.1
 blinker==1.5
 boto3==1.26.32
@@ -376,7 +376,7 @@ mongomock==4.1.2
 monotonic==1.6
 more-itertools==8.14.0
 moreorless==0.4.0
-moto==4.0.11
+moto==4.0.12
 msal-extensions==1.0.0
 msal==1.20.0
 msgpack==1.0.4
@@ -494,7 +494,7

[GitHub] [airflow] vchiapaikeo commented on issue #28393: Webserver reports "ImportError: Module "airflow.providers.google.cloud.operators.mlengine" does not define a "AIPlatformConsoleLink" attribute



vchiapaikeo commented on issue #28393:
URL: https://github.com/apache/airflow/issues/28393#issuecomment-1356852316

   Hey @uranusjr , I'm happy to remove this from the config but modifying the 
pre-commit hook is a bit outside my wheelhouse. Is it okay if we scope the 
pre-commit hook edits out to another issue instead?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on pull request #28442: Re-enable azure service bus on ARM as it now builds cleanly



Taragolis commented on PR #28442:
URL: https://github.com/apache/airflow/pull/28442#issuecomment-1356851154

   Oh, I just repeat step which probably most of user do first, just try to 
install it by `pip` into the regular image.
   
   Some of them after would read documentation and found this info, which 
really easy to find 
https://www.google.com/search?q=airflow+docker+install+python+packages, other 
would open issue, discussion, question on SO or into the Slack.
   
   Another problem for users what packages they need to install for build it, 
so that is why I suggest to create per-provider information about each 
extra/optional dependency (I could do that)
   
   > That would be - however - backwards incompatible for x86 users - because 
installing the provider will not pull azure-service-bus as requirement, so I am 
not too sure (we could likely come with some mixed approach where we have != 
aarch64 as "core" requirement and then have an extra to install it without that 
limitation - but I feel that would be rather cumbersome.
   
   I think better to keep compatibility and add as extra section even if it 
looks dumb. I bet that x86 main users now and for ARM we just have experimental 
support


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj

boring-cyborg[bot] commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1356850980

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vchiapaikeo opened a new pull request, #28444: Fix GCSToBigQueryOperator not respecting schema_obj



vchiapaikeo opened a new pull request, #28444:
URL: https://github.com/apache/airflow/pull/28444

   
   
   GCSToBigQueryOperator allows multiple ways to specify schema of the BigQuery 
table:
   
   1. Setting autodetect == True
   1. Setting schema_fields directly with autodetect == False
   1. Setting a schema_object and optionally a schema_object_bucket with 
autodetect == False
   
   This third method seems to be broken in the latest provider version (8.6.0) 
and will always result in this error:
   
   ```
   [2022-12-16, 21:06:18 UTC] {taskinstance.py:1772} ERROR - Task failed with 
exception
   Traceback (most recent call last):
 File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py",
 line 395, in execute
   self.configuration = self._check_schema_fields(self.configuration)
 File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py",
 line 524, in _check_schema_fields
   raise RuntimeError(
   RuntimeError: Table schema was not found. Set autodetect=True to 
automatically set schema fields from source objects or pass schema_fields 
explicitly
   ```
   
   The reason for this is because [this 
block](https://github.com/apache/airflow/blob/25bdbc8e6768712bad6043618242eec9c6632618/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py#L318-L320)
 where `if self.schema_object and self.source_format != "DATASTORE_BACKUP":`. 
fails to set self.schema_fields. It only sets the local variable, 
schema_fields. When self._check_schema_fields is subsequently called 
[here](https://github.com/apache/airflow/blob/25bdbc8e6768712bad6043618242eec9c6632618/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py#L395),
 we enter the [first 
block](https://github.com/apache/airflow/blob/25bdbc8e6768712bad6043618242eec9c6632618/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py#L523-L528)
 because autodetect is false and schema_fields is not set.
   
   This PR sets the instance variable, self.schema_fields when the user passes 
in a schema_obj. Additionally, it uses self.schema_object_bucket instead of the 
erroneous self.bucket.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vchiapaikeo commented on issue #28441: GCSToBigQueryOperator fails when schema_object is specified without schema_fields



vchiapaikeo commented on issue #28441:
URL: https://github.com/apache/airflow/issues/28441#issuecomment-1356851027

   Sure @eladkal - this is here: https://github.com/apache/airflow/pull/28444


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request, #28443: Re-enable Plyvel on ARM as it now builds cleanly



potiuk opened a new pull request, #28443:
URL: https://github.com/apache/airflow/pull/28443

   Previously, Plyvel had to be disabled in order to get ARM compatibility (it 
did not have
   binary wheels released and it failed to compile cleanly on debian). But the 
last problem is fixed now, so we can re-enable it for ARM.
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28442: Re-enable azure service bus on ARM as it now builds cleanly



potiuk commented on PR #28442:
URL: https://github.com/apache/airflow/pull/28442#issuecomment-1356844403

   > Hmmm.. I've unable to install it into the regular airflow image
   
   We have a number of requirements that require customising image rather than 
extending it. We have good instructions on it including how to customize image 
(if you want optimal size of the image):
   https://airflow.apache.org/docs/docker-stack/build.html#extending-the-image
   
   Or how to add a dependcy that requires compilation (I guess if you follow 
it, you will be able to install it):
   
   
https://airflow.apache.org/docs/docker-stack/build.html#example-when-you-add-packages-requiring-compilation
   
   > Maybe better keep requirement without any changes for now, create 
additional extra where we do not limit platform and later I could investigate 
and add into the Azure provider documentation about additional requirements for 
install it on Linux ARM.
   
   I think this is rather fine when a package has no wheel to expect it is 
going to be compiled - and when it compiles cleanly without too much of a 
hassle, I think it's perfectly fine. We have a number of other deps and 
providers that expect the "customize/add build-essentials" approach. And this 
one is only in case of ARM so I am not too worried - someone who wants to 
install azure provider on ARM is anyhow walking an experimental path and should 
be well aware of what they are doing.
   
   But maybe indeed in case of Azure we should not make it a requirement and 
make it optional extra of the provider - same as plyvel for google provider? 
   
   That would be - however - backwards incompatible for AMD users - because 
installing the provider will not pull azure-service-bus as requirement, so I am 
not too sure (we could likely come with some mixed approach where we have != 
aarch64 as "core" requirement and then have an extra to install it without that 
limitation - but I feel that would be rather cumbersome. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on pull request #28442: Re-enable azure service bus on ARM as it now builds cleanly



Taragolis commented on PR #28442:
URL: https://github.com/apache/airflow/pull/28442#issuecomment-1356842242

   Hmmm.. I've unable to install it into the regular airflow image
   
   ```console
   ❯ docker run -it --rm apache/airflow:2.5.0-python3.9 bash
   
   airflow@7e0bc7525889:/opt/airflow$ uname -a
   Linux 7e0bc7525889 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 
2021 aarch64 GNU/Linux
   
   airflow@7e0bc7525889:/opt/airflow$ pip install azure-servicebus
   Defaulting to user installation because normal site-packages is not writeable
   Collecting azure-servicebus
 Downloading azure_servicebus-7.8.1-py3-none-any.whl (210 kB)
 210.4/210.4 kB 2.3 MB/s eta 
0:00:00
   Requirement already satisfied: azure-core<2.0.0,>=1.14.0 in 
/home/airflow/.local/lib/python3.9/site-packages (from azure-servicebus) 
(1.26.1)
   Requirement already satisfied: typing-extensions>=3.7.4.3 in 
/home/airflow/.local/lib/python3.9/site-packages (from azure-servicebus) (4.4.0)
   Requirement already satisfied: six>=1.11.0 in 
/home/airflow/.local/lib/python3.9/site-packages (from azure-servicebus) 
(1.16.0)
   Requirement already satisfied: isodate>=0.6.0 in 
/home/airflow/.local/lib/python3.9/site-packages (from azure-servicebus) (0.6.1)
   Requirement already satisfied: azure-common~=1.1 in 
/home/airflow/.local/lib/python3.9/site-packages (from azure-servicebus) 
(1.1.28)
   Requirement already satisfied: msrest<2.0.0,>=0.6.17 in 
/home/airflow/.local/lib/python3.9/site-packages (from azure-servicebus) (0.7.1)
   Collecting uamqp<2.0.0,>=1.5.1
 Downloading uamqp-1.6.3.tar.gz (4.4 MB)
 4.4/4.4 MB 3.7 MB/s eta 0:00:00
 Installing build dependencies ... done
 Getting requirements to build wheel ... done
 Preparing metadata (pyproject.toml) ... done
   Requirement already satisfied: requests>=2.18.4 in 
/home/airflow/.local/lib/python3.9/site-packages (from 
azure-core<2.0.0,>=1.14.0->azure-servicebus) (2.28.1)
   Requirement already satisfied: certifi>=2017.4.17 in 
/home/airflow/.local/lib/python3.9/site-packages (from 
msrest<2.0.0,>=0.6.17->azure-servicebus) (2022.9.24)
   Requirement already satisfied: requests-oauthlib>=0.5.0 in 
/home/airflow/.local/lib/python3.9/site-packages (from 
msrest<2.0.0,>=0.6.17->azure-servicebus) (1.3.1)
   Requirement already satisfied: urllib3<1.27,>=1.21.1 in 
/home/airflow/.local/lib/python3.9/site-packages (from 
requests>=2.18.4->azure-core<2.0.0,>=1.14.0->azure-servicebus) (1.26.13)
   Requirement already satisfied: charset-normalizer<3,>=2 in 
/home/airflow/.local/lib/python3.9/site-packages (from 
requests>=2.18.4->azure-core<2.0.0,>=1.14.0->azure-servicebus) (2.1.1)
   Requirement already satisfied: idna<4,>=2.5 in 
/home/airflow/.local/lib/python3.9/site-packages (from 
requests>=2.18.4->azure-core<2.0.0,>=1.14.0->azure-servicebus) (3.4)
   Requirement already satisfied: oauthlib>=3.0.0 in 
/home/airflow/.local/lib/python3.9/site-packages (from 
requests-oauthlib>=0.5.0->msrest<2.0.0,>=0.6.17->azure-servicebus) (3.2.2)
   Building wheels for collected packages: uamqp
 Building wheel for uamqp (pyproject.toml) ... error
 error: subprocess-exited-with-error
 
 × Building wheel for uamqp (pyproject.toml) did not run successfully.
 │ exit code: 1
 ╰─> [101 lines of output]
 running bdist_wheel
 running build
 running build_py
 creating build
 creating build/lib.linux-aarch64-cpython-39
 creating build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/mgmt_operation.py -> 
build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/message.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/address.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/errors.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/session.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/receiver.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/utils.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/__init__.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/types.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/connection.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/compat.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/sender.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/constants.py -> build/lib.linux-aarch64-cpython-39/uamqp
 copying uamqp/client.py -> build/lib.linux-aarch64-cpython-39/uamqp
 creating build/lib.linux-aarch64-cpython-39/uamqp/authentication
 copying uamqp/authentication/common.py -> 
build/lib.linux-aarch64-cpython-39/uamqp/authentication
 copying uamqp/authentication/cbs_auth.py ->

[GitHub] [airflow] potiuk commented on pull request #28442: Re-enable azure service bus on ARM as it now builds cleanly



potiuk commented on PR #28442:
URL: https://github.com/apache/airflow/pull/28442#issuecomment-1356840195

   Actually - when I tried it on ARM now, the azure-service-bus package built 
cleanly on ARM without any setup and special libraries. I think it is quite ok 
to add it - we have quite a few packages that are built during installation on 
ARM, and this generally is not a big problem as long as the packages CAN be 
cleanly built without special workarounds or libraries.
   
   Similarly plyvel - plyvel is optional dependency of Google Package and as 
long as we can build it cleanly (will try your solution from #28432 @Taragolis) 
I will re-add it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request, #28442: Re-enable azure service bus on ARM as it now builds cleanly



potiuk opened a new pull request, #28442:
URL: https://github.com/apache/airflow/pull/28442

   Previously, the Azure Service Bus had to be disabled in order to get ARM 
compatibility (it failed to build cleanly as uampq did not have binary wheels 
released and they failed to compile cleanly on debian). But the last problem is 
fixed now, so we can re-enable it for ARM.
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #28441: GCSToBigQueryOperator fails when schema_object is specified without schema_fields



eladkal commented on issue #28441:
URL: https://github.com/apache/airflow/issues/28441#issuecomment-1356838736

   The previous issue is 2 years old so I'm not sure if the description there 
is out of date?
   In any case the issue is assigned to you. waiting for your PR :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vchiapaikeo commented on issue #28441: GCSToBigQueryOperator fails when schema_object is specified without schema_fields



vchiapaikeo commented on issue #28441:
URL: https://github.com/apache/airflow/issues/28441#issuecomment-1356838361

   @eladkal , I don't think this is a duplicate. This issue seems like a bug 
because we are not setting the instance variable, self.schema_fields and are 
instead setting the local variable, schema_fields.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: Fix discoverability of tests for ARM in Breeze (#28432)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 2a78f50b36 Fix discoverability of tests for ARM in Breeze (#28432)
2a78f50b36 is described below

commit 2a78f50b36eb7d0e4589633d12458eabbf82418d
Author: Jarek Potiuk 
AuthorDate: Sun Dec 18 18:03:46 2022 +0100

Fix discoverability of tests for ARM in Breeze (#28432)

Breeze in case of ARM processor lacks support for several components
(because they do not have supported ARM binaries available):

* MySQL
* MSSQL
* LevelDB
* Azure Service Bus

When you try to attempt to run pytest on a group of tests that import
one of those, the collection failed and none of the tests could run
even if some of them could.

This change uses pytest's skip on a module level and local imports
in case the tests are inter-mixed with other tests in the same module
to avoid import errors during collection.

The try/except pattern over pytest.importorskip is preferred because
we are using try/except in a number of other cases and we are pretty
familiar with similar pattern and importorskipi has a bit unexpected
behaviour (it returns imported module and you do not see the usual
`import `. Also in our case we often wrap more than one
import in one try/except (and it would lead to a duplicating messages
to print really.

We also add a separate command in ci to just perform a collection of
tests and see if all tests are collectable after uninstalling all
those libraries. This would prevent the problems from reapparing.

Isort fixes are implemented for recently relesed isort version
---
 .github/workflows/ci.yml   |  6 ++-
 scripts/in_container/test_arm_pytest_collection.py | 53 ++
 tests/operators/test_generic_transfer.py   |  6 ++-
 .../apache/hive/transfers/test_mssql_to_hive.py| 12 +++--
 .../apache/hive/transfers/test_mysql_to_hive.py|  8 +++-
 .../cloud/transfers/test_bigquery_to_mssql.py  |  9 +++-
 .../google/cloud/transfers/test_mssql_to_gcs.py|  8 +++-
 .../google/cloud/transfers/test_mysql_to_gcs.py| 11 +++--
 .../google/cloud/triggers/test_cloud_build.py  |  2 -
 .../providers/google/leveldb/hooks/test_leveldb.py |  7 ++-
 .../google/leveldb/operators/test_leveldb.py   | 11 -
 tests/providers/microsoft/azure/hooks/test_asb.py  |  8 +++-
 .../microsoft/azure/operators/test_asb.py  |  6 ++-
 .../providers/microsoft/mssql/hooks/test_mssql.py  |  6 ++-
 .../microsoft/mssql/operators/test_mssql.py| 10 +++-
 tests/providers/mysql/hooks/test_mysql.py  | 11 -
 .../mysql/transfers/test_vertica_to_mysql.py   |  8 +++-
 .../cloud/bigquery/example_bigquery_to_mssql.py|  8 +++-
 .../google/cloud/gcs/example_mssql_to_gcs.py   |  9 +++-
 .../google/cloud/gcs/example_mysql_to_gcs.py   |  9 +++-
 .../providers/google/leveldb/example_leveldb.py| 10 +++-
 .../microsoft/azure/example_azure_service_bus.py   | 30 +++-
 .../providers/microsoft/mssql/example_mssql.py | 10 +++-
 23 files changed, 212 insertions(+), 46 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 532161aef5..7cf54ccb7b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -816,8 +816,10 @@ jobs:
 uses: ./.github/actions/prepare_breeze_and_image
   - name: "Migration Tests: 
${{matrix.python-version}}:${{needs.build-info.outputs.test-types}}"
 uses: ./.github/actions/migration_tests
-  - name: "Tests: 
${{matrix.python-version}}:${{needs.build-info.outputs.test-types}} 
(w/Kerberos)"
+  - name: "Tests: 
${{matrix.python-version}}:${{needs.build-info.outputs.test-types}}"
 run: breeze testing tests --run-in-parallel
+  - name: "Tests ARM Pytest collection: ${{matrix.python-version}}"
+run: breeze shell "python 
/opt/airflow/scripts/in_container/test_arm_pytest_collection.py"
   - name: "Post Tests: 
${{matrix.python-version}}:${{needs.build-info.outputs.test-types}}"
 uses: ./.github/actions/post_tests
 
@@ -941,6 +943,8 @@ jobs:
 uses: ./.github/actions/migration_tests
   - name: "Tests: 
${{matrix.python-version}}:${{needs.build-info.outputs.test-types}}"
 run: breeze testing tests --run-in-parallel
+  - name: "Tests ARM Pytest collection: ${{matrix.python-version}}"
+run: breeze shell "python 
/opt/airflow/scripts/in_container/test_arm_pytest_collection.py"
   - name: "Post Tests: 
${{matrix.python-version}}:${{needs.build-info.outputs.test-types}}"
 uses: ./.github/actions/post_tests
 
diff --git a/scripts/in_container/test_arm_pytest_collection.py 
b/scripts/in_container/test_arm_pytest_collection.py
new file

[GitHub] [airflow] potiuk merged pull request #28432: Fix discoverability of tests for ARM in Breeze



potiuk merged PR #28432:
URL: https://github.com/apache/airflow/pull/28432


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28432: Fix discoverability of tests for ARM in Breeze



potiuk commented on PR #28432:
URL: https://github.com/apache/airflow/pull/28432#issuecomment-1356837794

   All looks good. Merging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28432: Fix discoverability of tests for ARM in Breeze



potiuk commented on PR #28432:
URL: https://github.com/apache/airflow/pull/28432#issuecomment-1356837188

   > Actually this is a "grey zone": uamqp still do not have wheel packages for 
[aarch64](https://pypi.org/project/uamqp/#files) - open issue in Github: 
https://github.com/Azure/azure-uamqp-python/issues/349, so end users need to 
pre-install some dev packages for install it from tar.gz distribution,
   
   Yep. I just read the discusion in the linked issue in the code. I think I 
will wait with that for the official wheel. This is not blocking for anyone.  
If they will build the package, the azure-service-bus willl start working for 
them - so this is just a matter of automatically pulling it as dependency.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Taragolis commented on a diff in pull request #28432: Fix discoverability of tests for ARM in Breeze



Taragolis commented on code in PR #28432:
URL: https://github.com/apache/airflow/pull/28432#discussion_r1051632929


##
tests/providers/microsoft/azure/hooks/test_asb.py:
##
@@ -19,8 +19,12 @@
 from unittest import mock
 
 import pytest
-from azure.servicebus import ServiceBusClient, ServiceBusMessage, 
ServiceBusMessageBatch
-from azure.servicebus.management import ServiceBusAdministrationClient
+
+try:
+from azure.servicebus import ServiceBusClient, ServiceBusMessage, 
ServiceBusMessageBatch
+from azure.servicebus.management import ServiceBusAdministrationClient
+except ImportError:
+pytest.skip("Azure Service Bus not available", allow_module_level=True)

Review Comment:
   Actually this is a "grey zone": `uamqp` still do not have wheel packages for 
[aarch64](https://pypi.org/project/uamqp/#files) - open issue in Github: 
https://github.com/Azure/azure-uamqp-python/issues/349, so end users need to 
pre-install some dev packages for install it from tar.gz distribution, 
   
   So if we change requirements for package then someone who intend to use 
airflow on ARM (yeah this is experimental) faced with issues to install entire 
provider package.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated (f115b207bc -> efaac217e9)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


from f115b207bc fIx isort problems introduced by recent isort release 
(#28434)
 add efaac217e9 Fix output buffering for `breeze testing test` (#28433)

No new revisions were added by this update.

Summary of changes:
 scripts/in_container/filter_out_warnings.py | 2 ++
 1 file changed, 2 insertions(+)

[GitHub] [airflow] potiuk merged pull request #28433: Fix output buffering for `breeze testing test`



potiuk merged PR #28433:
URL: https://github.com/apache/airflow/pull/28433


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28433: Fix output buffering for `breeze testing test`



potiuk commented on PR #28433:
URL: https://github.com/apache/airflow/pull/28433#issuecomment-1356833772

   Static checks passed. Merging/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #28432: Fix discoverability of tests for ARM in Breeze



potiuk commented on code in PR #28432:
URL: https://github.com/apache/airflow/pull/28432#discussion_r1051630643


##
tests/providers/microsoft/azure/hooks/test_asb.py:
##
@@ -19,8 +19,12 @@
 from unittest import mock
 
 import pytest
-from azure.servicebus import ServiceBusClient, ServiceBusMessage, 
ServiceBusMessageBatch
-from azure.servicebus.management import ServiceBusAdministrationClient
+
+try:
+from azure.servicebus import ServiceBusClient, ServiceBusMessage, 
ServiceBusMessageBatch
+from azure.servicebus.management import ServiceBusAdministrationClient
+except ImportError:
+pytest.skip("Azure Service Bus not available", allow_module_level=True)

Review Comment:
   Ah. Maybe it has been fixed already. Let me try it as well. Good points :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #28432: Fix discoverability of tests for ARM in Breeze



potiuk commented on code in PR #28432:
URL: https://github.com/apache/airflow/pull/28432#discussion_r1051630531


##
tests/system/providers/google/leveldb/example_leveldb.py:
##
@@ -23,8 +23,16 @@
 import os
 from datetime import datetime
 
+import pytest
+
 from airflow import models
-from airflow.providers.google.leveldb.operators.leveldb import LevelDBOperator
+from airflow.exceptions import AirflowOptionalProviderFeatureException
+
+try:
+from airflow.providers.google.leveldb.operators.leveldb import 
LevelDBOperator
+except AirflowOptionalProviderFeatureException:
+pytest.skip("LevelDB not available", allow_module_level=True)

Review Comment:
   Yeah we can attempt to do it as next step. The difference is that the first 
one was needed for image building that's why aarch exclusion - the second is 
just extra specification for google provider (and those two might not be 
synchronized indeed). But if we can build it in the way you described, that 
would be better than excluding it. I will try it in a moment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28300: Add Public Interface description to Airflow documentation



potiuk commented on PR #28300:
URL: https://github.com/apache/airflow/pull/28300#issuecomment-1356830770

   Rebased to apply static check fixes with isort.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28433: Fix output buffering for `breeze testing test`



potiuk commented on PR #28433:
URL: https://github.com/apache/airflow/pull/28433#issuecomment-1356830183

   Rebased to account for static check fixes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch main updated: fIx isort problems introduced by recent isort release (#28434)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new f115b207bc fIx isort problems introduced by recent isort release 
(#28434)
f115b207bc is described below

commit f115b207bc844c10569b2df6fc9acfa32a3c7f41
Author: Jarek Potiuk 
AuthorDate: Sun Dec 18 17:04:48 2022 +0100

fIx isort problems introduced by recent isort release (#28434)

The recent isort changed their mind on sorting the imports. This
change follows the change and bumps isort to latest released
version (isort has no install_requires on its own so bumping
min version has no effect on other dependencies)

This change adds a number of isort:skip_file, isort:off, isort:skips
in order to handle a very annoying bug in isort, that no matter how
much you try, it sometimes treat "known first party" packages
differently - depending on how many files it processes at a time.

We should be able to restore it after this bug is fixed:
https://github.com/PyCQA/isort/issues/2045

This change also updates the common.sql API to skip them from isort
for the very same reason (depending on how many files are modified,
the isort order might change.
---
 .pre-commit-config.yaml| 28 +++--
 airflow/providers/common/sql/hooks/sql.pyi |  4 ++
 airflow/providers/common/sql/operators/sql.pyi |  4 ++
 airflow/providers/common/sql/sensors/sql.pyi   |  4 ++
 docker_tests/test_docker_compose_quick_start.py|  3 ++
 .../test_examples_of_prod_image_building.py|  3 ++
 docker_tests/test_prod_image.py|  3 ++
 docs/build_docs.py | 11 --
 docs/exts/docs_build/dev_index_generator.py|  3 ++
 docs/exts/docs_build/errors.py |  4 +-
 docs/publish_docs.py   |  3 ++
 kubernetes_tests/test_kubernetes_executor.py   |  2 +-
 kubernetes_tests/test_other_executors.py   |  2 +-
 pyproject.toml |  1 +
 .../pre_commit_update_common_sql_api_stubs.py  | 46 --
 setup.py   |  5 ++-
 16 files changed, 95 insertions(+), 31 deletions(-)

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 7506249602..8b3f4c732d 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -146,6 +146,21 @@ repos:
   - --fuzzy-match-generates-todo
 files: >
   
\.cfg$|\.conf$|\.ini$|\.ldif$|\.properties$|\.readthedocs$|\.service$|\.tf$|Dockerfile.*$
+  - repo: local
+hooks:
+  - id: update-common-sql-api-stubs
+name: Check and update common.sql API stubs
+entry: 
./scripts/ci/pre_commit/pre_commit_update_common_sql_api_stubs.py
+language: python
+files: 
^scripts/ci/pre_commit/pre_commit_update_common_sql_api\.py|^airflow/providers/common/sql/.*\.pyi?$
+additional_dependencies: ['rich>=12.4.4', 'mypy==0.971', 
'black==22.3.0', 'jinja2']
+pass_filenames: false
+require_serial: true
+  - repo: https://github.com/PyCQA/isort
+rev: 5.11.2
+hooks:
+  - id: isort
+name: Run isort to sort imports in Python files
   # Keep version of black in sync wit blacken-docs, pre-commit-hook-names, 
update-common-sql-api-stubs
   - repo: https://github.com/psf/black
 rev: 22.3.0
@@ -233,11 +248,6 @@ repos:
 entry: yamllint -c yamllint-config.yml --strict
 types: [yaml]
 exclude: 
^.*init_git_sync\.template\.yaml$|^.*airflow\.template\.yaml$|^chart/(?:templates|files)/.*\.yaml$|openapi/.*\.yaml$|^\.pre-commit-config\.yaml$|^airflow/_vendor/
-  - repo: https://github.com/PyCQA/isort
-rev: 5.10.1
-hooks:
-  - id: isort
-name: Run isort to sort imports in Python files
   - repo: https://github.com/pycqa/pydocstyle
 rev: 6.1.1
 hooks:
@@ -396,14 +406,6 @@ repos:
 language: python
 files: ^docs
 pass_filenames: false
-  - id: update-common-sql-api-stubs
-name: Check and update common.sql API stubs
-entry: 
./scripts/ci/pre_commit/pre_commit_update_common_sql_api_stubs.py
-language: python
-files: 
^scripts/ci/pre_commit/pre_commit_update_common_sql_api\.py|^airflow/providers/common/sql/.*\.pyi?$
-additional_dependencies: ['rich>=12.4.4', 'mypy==0.971', 
'black==22.3.0']
-pass_filenames: false
-require_serial: true
   - id: check-pydevd-left-in-code
 language: pygrep
 name: Check for pydevd debug statements accidentally left
diff --git a/airflow/providers/common/sql/hooks/sql.pyi 
b/airflow/providers/common/sql/hooks/sql.pyi
index 718f09da7f..30d8eef488 100644
--- a/airflow/providers/common/sql/hooks/sql.pyi
+++

[GitHub] [airflow] potiuk merged pull request #28434: Fix isort problems introduced by recent isort release



potiuk merged PR #28434:
URL: https://github.com/apache/airflow/pull/28434


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28434: Fix isort problems introduced by recent isort release



potiuk commented on PR #28434:
URL: https://github.com/apache/airflow/pull/28434#issuecomment-1356828189

   Just flaky test failed in MSSQL. Merging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #28441: GCSToBigQueryOperator fails when schema_object is specified without schema_fields



eladkal commented on issue #28441:
URL: https://github.com/apache/airflow/issues/28441#issuecomment-1356827403

   Duplicate of https://github.com/apache/airflow/issues/12329


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #28436: Log FileTaskHandler to work with multi_namespace_mode when running using KubernetesExecutor