kevinalh opened a new issue, #38006:
URL: https://github.com/apache/airflow/issues/38006
### Apache Airflow version
2.8.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Was doing an upgrade from Airflow 2.2.2 to 2.8.1, and encountered the error
`ValueError: non-default argument follows default argument` on my TaskFlow
decorated operators. Had to go through the Airflow code to understand what was
going on.
The general shape of my tasks is:
```python
@task(multiple_outputs=True)
def decorated_task(
inputs: list[dict],
logical_date: str,
project_id: str,
bucket_name: str,
) -> dict[str, str]:
# Implementation details
return {}
```
called from a DAG like:
```python
@dag(
dag_id=dag_id,
# ...
default_args={
"depends_on_past": False,
"retries": 3,
"retry_delay": timedelta(hours=1),
},
)
def decorated_dag():
# ...
task_date = "{{ (dag_run.logical_date - macros.timedelta(days=1)) |
ds_nodash }}"
# ...
decorated_task_result = decorated_task(
inputs=inputs,
logical_date=task_date,
project_id=PROJECT_ID,
bucket_name=BUCKET_NAME,
)
```
The error comes with a stack trace like follows:
```
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/dag.py",
line 3944, in factory
f(**f_kwargs)
File "/usr/local/airflow/dags/dag_file.py", line 160, in decorated_dag
decorated_task_result = decorated_task(
^^^^^^^^^^^^^^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/decorators/base.py",
line 366, in __call__
op = self.operator_class(
^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py",
line 437, in apply_defaults
result = func(self, **kwargs, default_args=default_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/decorators/python.py",
line 52, in __init__
super().__init__(
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py",
line 437, in apply_defaults
result = func(self, **kwargs, default_args=default_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/decorators/base.py",
line 217, in __init__
signature = signature.replace(parameters=parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/inspect.py", line 3052, in replace
return type(self)(parameters,
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/inspect.py", line 3008, in __init__
raise ValueError(msg)
ValueError: non-default argument follows default argument
```
The problem also happens with 2.8.2.
As the trace mentions, the problem is in this part of the code of the
`DecoratedOperator` class in `decorators/base.py`:
```python
# Check the decorated function's signature. We go through the argument
# list and "fill in" defaults to arguments that are known context keys,
# since values for those will be provided when the task is run. Since
# we're not actually running the function, None is good enough here.
signature = inspect.signature(python_callable)
parameters = [
param.replace(default=None) if param.name in KNOWN_CONTEXT_KEYS else
param
for param in signature.parameters.values()
]
signature = signature.replace(parameters=parameters)
```
Since the `logical_date` parameter has the name of a known context key,
giving it the `None` default when the rest of the parameters don't have a
default generates the error.
### What you think should happen instead?
Two options come to mind:
1. Make the error message clearer so that it's explicitly mentioned that
context variables should go _after_ other parameters in your TaskFlow function
signatures. We should also include this in the documentation.
2. Allow for this behavior by changing code. We could simply do this by
making the known context parameters with the `None` default go to the end.
### How to reproduce
Doing a brand-new local installation of Airflow with
```
mkdir test_airflow
cd test_airflow
python -m venv ./.venv
source ./.venv/bin/activate.fish
pip install "apache-airflow==2.8.2" --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-2.8.2/constraints-3.11.txt";
```
and adding a simple DAG:
```python
from airflow.decorators import task, dag
@task
def run_task(logical_date: str, input: str) -> str:
return f"{logical_date}, {input}"
@dag(tags=["test"])
def run_dag():
run_task(input="test_input", logical_date="{{ logical_date }}")
run_dag()
```
reproduces the issue. Putting `logical_date` as the second argument of
`run_task` fixes it, as expected.
### Operating System
macOS Sonoma 14.3.1
### Versions of Apache Airflow Providers
Only the default ones, this isn't relevant.
### Deployment
Virtualenv installation
### Deployment details
Was able to reproduce it with a minimal local virtualenv installation, but
originally found the issue in the MWAA local runner.
### Anything else?
After figuring it out I posted about it on StackOverflow in case someone
else encounters the problem:
https://stackoverflow.com/questions/78131051/valueerror-non-default-argument-follows-default-argument-when-upgrading-from
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
