Re: [I] KubernetesPodOperator dry_run failure [airflow]
nevcohen commented on issue #45812: URL: https://github.com/apache/airflow/issues/45812#issuecomment-2622812645 If I'm not mistaken, the purpose of `dry run` is to simulate running in a real environment, so you must give a credential (or kube config), if not, then where do you want it to run? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] KubernetesPodOperator dry_run failure [airflow]
potiuk closed issue #45916: KubernetesPodOperator dry_run failure URL: https://github.com/apache/airflow/issues/45916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] KubernetesPodOperator dry_run failure [airflow]
nathadfield commented on issue #45916: URL: https://github.com/apache/airflow/issues/45916#issuecomment-2607730258 @Arjit-Sharma001 Given that the current version of the cncf provider is [10.1](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/changelog.html), there have clearly been many bug fixes and improvements since 4.3.0. There will not be any fixes to old versions so, if you are able to, can you test your problem against a newer version? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] KubernetesPodOperator dry_run failure [airflow]
Arjit-Sharma001 opened a new issue, #45916:
URL: https://github.com/apache/airflow/issues/45916
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes 4.3.0
### Apache Airflow version
2.3.4
### Operating System
Linux
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
We are upgrading from apache-airflow-providers-cncf-kubernetes version 3.0.0
to 4.3.0, gradually moving through releases.
Our current setup includes a custom script that, during the Docker image
build for Airflow, tests all DAGs and tasks in dry_run mode. This script is
essential for detecting issues such as Python syntax errors, DAG cycles,
duplicate tasks, import errors, and template rendering issues.
This process was functioning seamlessly with our existing Airflow version.
However, as part of our upgrade to a newer version of Airflow (and its
providers), we encountered an issue with the KubernetesPodOperator's dry_run
functionality after addressing several other compatibility challenges.
Problem Description
The new dry_run implementation introduced in [this
commit](https://github.com/apache/airflow/commit/d56ff765e15f9fcd582bc6d1ec0e83b0fedf476a)
invokes the build_pod_request_obj method. This method calls the
self.hook.is_in_cluster property, which attempts to construct a Kubernetes API
client object and requires kube client credentials.
```
pod.metadata.labels.update(
{
'airflow_version': airflow_version.replace('+', '-'),
'airflow_kpo_in_cluster': str(self.hook.is_in_cluster),
}
)
```
The issue lies in the is_in_cluster property:
```
@property
def is_in_cluster(self):
"""Expose whether the hook is configured with ``load_incluster_config``
or not"""
if self._is_in_cluster is not None:
return self._is_in_cluster
self.api_client # so we can determine if we are in_cluster or not
return self._is_in_cluster
```
This behavior causes the dry_run to fail in an isolated test environment
without valid kube config or credentials.
Error Traceback
When running in this environment, the following error occurs:
`kubernetes.config.config_exception.ConfigException: Invalid kube-config
file. No configuration found.
`
Desired Behavior
We want to continue using dry_run to test DAGs without requiring Kubernetes
credentials or kube config. The tests don't need to be fully accurate but
should bypass unnecessary configurations.
Proposed Solutions
Introduce an Environment Variable
Add an environment variable to bypass the airflow_kpo_in_cluster label
setting in dry_run mode when explicitly requested by the user.
Modify build_pod_request_obj Signature
Change the method signature of build_pod_request_obj to include a dry_run:
bool = False keyword argument.
In the KubernetesPodOperator.dry_run method, invoke
build_pod_request_obj(dry_run=True).
Skip setting the airflow_kpo_in_cluster label if dry_run=True.
Implement Both Options
Combine both approaches to provide flexibility.
These changes would enable dry_run functionality to work in isolated
environments without requiring kube config or credentials while maintaining
compatibility with the operator's primary functionality.
### What you think should happen instead
The KubernetesPodOperator.dry_run should allow testing DAGs and tasks
without requiring Kubernetes credentials or kube config. This would ensure that
the dry_run functionality can execute in isolated or non-production
environments without unnecessary dependencies. It does not need to fully mimic
runtime behavior but should bypass features dependent on Kubernetes API calls,
such as the airflow_kpo_in_cluster label.
What Went Wrong
The new dry_run implementation attempts to evaluate self.hook.is_in_cluster,
which requires constructing a Kubernetes API client. This step fails when no
kube config or credentials are provided in the test environment, leading to a
ConfigException.
Supporting Evidence
Error Log Fragment
`kubernetes.config.config_exception.ConfigException: Invalid kube-config
file. No configuration found.
`
Root Cause
The build_pod_request_obj method in dry_run mode invokes
self.hook.is_in_cluster as part of adding metadata labels:
`'airflow_kpo_in_cluster': str(self.hook.is_in_cluster),`
This property requires access to Kubernetes credentials, which are not
available in the testing environment.
### How to reproduce
Set Up the Environment
Use Apache Airflow 2.10.4 and upgrade the
apache-airflow-providers-cncf-kubernetes package to version 4.3.0.
Ensure Kubernetes credentials (kube config) are not provided in
Re: [I] KubernetesPodOperator dry_run failure [airflow]
potiuk commented on issue #45812: URL: https://github.com/apache/airflow/issues/45812#issuecomment-2605030578 There are no "kubernetes provider maintainers" here. It's all "airflow" maintainers - anyone can comment here, and any maintainer can approve PR that is created. Since we are in the hottest part of Airlfow 3 building, it's not very likely that you will get more feedback than that, so creating a good PR with proposal how you would like to solve it is the best way to grab attention of maintainers who could approve it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] KubernetesPodOperator dry_run failure [airflow]
baryluk commented on issue #45812: URL: https://github.com/apache/airflow/issues/45812#issuecomment-2604891270 @potiuk Sure, I can, but I wanted to get some feedback from kubernetes provider maintainers first what would they prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] KubernetesPodOperator dry_run failure [airflow]
potiuk commented on issue #45812: URL: https://github.com/apache/airflow/issues/45812#issuecomment-2603264801 Feel free to attempt to fix it and provide PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] KubernetesPodOperator dry_run failure [airflow]
baryluk opened a new issue, #45812:
URL: https://github.com/apache/airflow/issues/45812
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes 4.3.0
### Apache Airflow version
2.3.4
### Operating System
Linux
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
n/a
### What happened
We are upgrading from apache-airflow-providers-cncf-kubernetes 3.0.0 to
4.3.0 (going slowly through releases).
We have a custom script, that during docker image build of our airflow,
tests all dags and all dag tasks in dry_run mode. Mostly to detect Python
syntax errors, dag cycles duplicate tasks, wrong imports, ntemplating errors
etc.
This was working all fine with our existing airflow, but we decided to
upgrade airflow to newer version, and that also means updating airflow
providers. After fixing bunch of other issues, I found the issues with
KubernetedPodOperator dry run.
New dry_run added in
https://github.com/apache/airflow/commit/d56ff765e15f9fcd582bc6d1ec0e83b0fedf476a
invokes `KubernetesPodOperator` `build_pod_request_obj()` method which has a
call to a property `self.hook.is_in_cluster`:
```python
pod.metadata.labels.update(
{
'airflow_version': airflow_version.replace('+', '-'),
'airflow_kpo_in_cluster': str(self.hook.is_in_cluster),
}
)
```
Unfortunately this property constructs a Kube API client object which
requires kube client config / credentials to work.
```python
@property
def is_in_cluster(self):
"""Expose whether the hook is configured with
``load_incluster_config`` or not"""
if self._is_in_cluster is not None:
return self._is_in_cluster
self.api_client # so we can determine if we are in_cluster or not
return self._is_in_cluster```
This causes dry_run to not able to execute in isolated test environment:
```
Traceback (most recent call last):
File "", line 97, in
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dag.py", line
2307, in cli
args.func(args, self)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/cli_parser.py",
line 51, in command
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/cli.py", line
99, in wrapper
return f(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py",
line 545, in task_test
ti.dry_run()
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py",
line 1815, in dry_run
self.task.dry_run()
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 607, in dry_run
pod = self.build_pod_request_obj()
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 595, in build_pod_request_obj
'airflow_kpo_in_cluster': str(self.hook.is_in_cluster),
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py",
line 283, in is_in_cluster
self.api_client # so we can determine if we are in_cluster or not
File "/usr/local/lib/python3.9/functools.py", line 993, in __get__
val = self.func(instance)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py",
line 291, in api_client
return self.get_conn()
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py",
line 239, in get_conn
config.load_kube_config(
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/config/kube_config.py",
line 808, in load_kube_config
loader = _get_kube_config_loader(
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/config/kube_config.py",
line 767, in _get_kube_config_loader
raise ConfigException(
kubernetes.config.config_exception.ConfigException: Invalid kube-config
file. No configuration found.
```
We would like to continue using dry_run, but be able to run it without
providing credentials or kube config. It does not need to be 100% accurate.
Two options:
* env var to bypass setting of `airflow_kpo_in_cluster` label in dry run
mode, if user requests to do so.
* never populate it in dry_run mode. (change signature of
`build_pod_request_obj` to have `dry_run: bool = False` kwarg and invoke it
with `dry_run=True` in KubernetesPodOperator.dry_run()` method.
(or both)
### What you thin
