This is an automated email from the ASF dual-hosted git repository.
kaxil pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new a9dd016df8a Validate Param date-time values again (#76561)
a9dd016df8a is described below
commit a9dd016df8a6068f3f81051584ce569f6732358f
Author: Shahar Epstein <[email protected]>
AuthorDate: Wed Nov 18 20:30:53 2026 +0200
Validate Param date-time values again (#76561)
* Validate Param date-time values again
jsonschema checks format="date-time" only when rfc3339-validator is
installed.
Airflow 3.0 removed that dependency, so a default install accepts any
string as
a date-time. CI did not catch this, because other dev dependencies install
the
package.
rfc3339-validator has had no release since 2021, and every alternative needs
the same custom check. Thus, Airflow now uses its own RFC 3339 check, which
needs only the standard library.
Public Dags compute values such as str(datetime.now()), which are ISO 8601
date-times but not RFC 3339. Strict validation would stop some of these Dags
from importing after an upgrade. Airflow 2 accepted these values with a
warning,
so Airflow accepts them again and leaves the string unchanged. A default
timezone would change the value that tasks receive. Values that are not
date-times still fail.
Stored Dags can hold a default that becomes invalid after the upgrade.
Without
a guard on the asset-triggered paths, that default makes the scheduler exit
until the Dag is parsed again. A backfill with no conf would also return a
500
and leave an open Backfill row.
Claude-Session: https://claude.ai/code/session_01BVx6g8A3doAQCz2buBCPa2
* Clarify date-time Param warnings and partition behavior
Only the Task SDK warns about deprecated date-time values, so the
docs said more than the code does. The partitioned-Dag test name and
scheduler comments also overstated that an invalid Dag cannot delay
others.
Claude-Session: https://claude.ai/code/session_01BVx6g8jDRinNACch6nQSs6
---
airflow-core/docs/core-concepts/params.rst | 9 +-
airflow-core/newsfragments/76561.significant.rst | 39 ++++++++
.../src/airflow/jobs/scheduler_job_runner.py | 107 ++++++++++++---------
airflow-core/src/airflow/models/backfill.py | 9 +-
.../src/airflow/serialization/definitions/param.py | 9 +-
airflow-core/tests/unit/jobs/test_scheduler_job.py | 79 +++++++++++++++
airflow-core/tests/unit/models/test_backfill.py | 26 ++++-
.../unit/serialization/definitions/test_param.py | 16 +++
.../src/airflow_shared/timezones/rfc3339.py | 58 +++++++++++
shared/timezones/tests/timezones/test_rfc3339.py | 76 +++++++++++++++
task-sdk/src/airflow/sdk/definitions/param.py | 22 ++++-
task-sdk/tests/task_sdk/definitions/test_param.py | 17 +++-
12 files changed, 406 insertions(+), 61 deletions(-)
diff --git a/airflow-core/docs/core-concepts/params.rst
b/airflow-core/docs/core-concepts/params.rst
index 78314893ec6..157a508ff50 100644
--- a/airflow-core/docs/core-concepts/params.rst
+++ b/airflow-core/docs/core-concepts/params.rst
@@ -245,7 +245,14 @@ The following features are supported in the Trigger UI
Form:
* | ``format="date"``: Generate a date-picker
| with calendar pop-up
* | ``format="date-time"``: Generate a date and
- | time-picker with calendar pop-up
+ | time-picker with calendar pop-up. Values should be
+ | RFC 3339 strings with a timezone offset, e.g.
+ | ``2024-01-01T00:00:00+00:00``. Other ISO 8601
+ | date-times are deprecated. Only the Task SDK
+ | warns about them, when it validates the value in
+ | Dag parsing or at task start. The API server and
+ | the scheduler accept them without a warning.
+ | Anything else is rejected.
* ``format="time"``: Generate a time-picker
* | ``format="duration"``: Generate a duration
| input field accepting ISO 8601 duration
diff --git a/airflow-core/newsfragments/76561.significant.rst
b/airflow-core/newsfragments/76561.significant.rst
new file mode 100644
index 00000000000..5891ba9e115
--- /dev/null
+++ b/airflow-core/newsfragments/76561.significant.rst
@@ -0,0 +1,39 @@
+Param values with ``format="date-time"`` are validated again
+
+Airflow 2 rejected a value that was not a date-time. It accepted an ISO 8601
date-time that was not RFC 3339,
+and showed a deprecation warning. Airflow 3.0 removed this fallback. It also
removed the ``rfc3339-validator``
+dependency, so a default install accepted any string. Airflow now validates
these values itself, with no extra
+dependency.
+
+A value is in one of three tiers:
+
+- Valid: an RFC 3339 string with a timezone offset, for example
``2024-01-01T00:00:00+00:00`` or
+ ``2024-01-01T00:00:00Z``.
+- Deprecated: an ISO 8601 date-time that is not RFC 3339, for example
``2024-01-01T00:00:00``,
+ ``2024-01-01 00:00:00+00:00`` or ``2024-01-01T00:00Z``. Airflow accepts the
value unchanged. Only the Task SDK
+ raises ``RemovedInAirflow4Warning``, when it validates the value in Dag
parsing or at task start. The API
+ server and the scheduler accept the value without a warning. A future major
release plans to reject these
+ values.
+- Not valid: any other value, for example ``2024-01-01``, ``{{ ds }}``, ``""``
or ``garbage``.
+
+A value that is not valid now has these effects:
+
+- A Dag with a schedule and a Dag-level default that is not valid fails to
import. The error is
+ ``Dag '<dag_id>' is not allowed to define a Schedule``.
+- A Dag with ``schedule=None`` and a Dag-level default that is not valid still
imports. A trigger with no
+ valid value is rejected.
+- A backfill of a Dag with a Dag-level default that is not valid, and no valid
``conf``, is rejected.
+- A task-level default that is not valid passes parsing. The task then fails
when it starts.
+- A Dag run with a value that is not valid in its stored ``conf`` fails when
its tasks start. This includes
+ queued runs, and old runs that you clear after the upgrade.
+
+**Migration:**
+
+- Use RFC 3339 values with a timezone offset for defaults of
``format="date-time"`` Params, at Dag level and
+ at task level.
+- Use RFC 3339 values with a timezone offset for these Params in ``conf`` that
you send through the API, the
+ CLI or ``TriggerDagRunOperator``.
+- Use ``.isoformat()`` on a timezone-aware ``datetime`` to get a valid value.
``str(datetime)`` gives a
+ space separator, which is deprecated.
+- ``{{ ts }}`` renders a valid value if the run has a logical date. ``{{
logical_date }}`` renders a
+ deprecated value.
diff --git a/airflow-core/src/airflow/jobs/scheduler_job_runner.py
b/airflow-core/src/airflow/jobs/scheduler_job_runner.py
index 1c47da65f9e..b4e341682d9 100644
--- a/airflow-core/src/airflow/jobs/scheduler_job_runner.py
+++ b/airflow-core/src/airflow/jobs/scheduler_job_runner.py
@@ -71,7 +71,7 @@ from airflow.callbacks.callback_requests import (
from airflow.configuration import conf
from airflow.dag_processing.bundles.base import BundleUsageTrackingManager
from airflow.dag_processing.maintenance import cleanup_processor_metadata
-from airflow.exceptions import DagNotFound, TaskNotFound,
UnknownExecutorException
+from airflow.exceptions import DagNotFound, ParamValidationError,
TaskNotFound, UnknownExecutorException
from airflow.executors import workloads
from airflow.executors.executor_loader import ExecutorLoader
from airflow.executors.workloads.types import TaskInstanceUuid
@@ -2521,12 +2521,12 @@ class SchedulerJobRunner(BaseJobRunner, LoggingMixin):
"""
Create Dag runs for pending :class:`AssetPartitionDagRun` rows whose
partition is satisfied.
- Returns the set of ``dag_id`` strings that received a new
partition-driven Dag run in this
- tick. The caller (:meth:`_create_dagruns_for_dags`) uses this set to
exclude the same Dags
- from the standard schedule-driven and asset-triggered creation paths
so a single Dag never
- gets two Dag runs for the same tick when it appears in more than one
creation path. We
- return ``dag_id`` strings rather than full Dag/DagRun objects because
the only downstream
- use is membership lookup, and a heavier return type would just be
discarded.
+ Returns the set of ``dag_id`` strings with a satisfied partition in
this tick, whether or not
+ creating the Dag run succeeded. The caller
(:meth:`_create_dagruns_for_dags`) uses this set to
+ exclude the same Dags from the standard schedule-driven and
asset-triggered creation paths so a
+ single Dag never gets two Dag runs for the same tick when it appears
in more than one creation
+ path. We return ``dag_id`` strings rather than full Dag/DagRun objects
because the only
+ downstream use is membership lookup, and a heavier return type would
just be discarded.
Asset deactivation freezes pending APDRs: when an asset becomes
inactive
(orphan — no Dag declares it any more), its ``PartitionedAssetKeyLog``
rows
@@ -2555,9 +2555,10 @@ class SchedulerJobRunner(BaseJobRunner, LoggingMixin):
# scheduling work isn't starved. Remaining APDRs drain across
subsequent ticks.
# FIFO is intentional: the oldest pending APDR fires first. A
persistently
# unsatisfiable APDR at the head (e.g. broken mapper, upstream that
will
- # never arrive) blocks newer ones until an operator removes it or fixes
- # the underlying mapper. We surface the stuck state rather than
silently
- # rotating past it.
+ # never arrive, a stored Dag whose Param default fails validation
until a
+ # re-parse marks the Dag stale) blocks newer ones until an operator
removes
+ # it or fixes the underlying mapper. We surface the stuck state rather
than
+ # silently rotating past it.
# `with_row_locks(skip_locked=True)` mirrors the sibling ADRQ claim
path:
# in HA two schedulers can otherwise both grab the same satisfied APDR
# and race the `created_dag_run_id` UPDATE, orphaning whichever DagRun
@@ -2721,21 +2722,26 @@ class SchedulerJobRunner(BaseJobRunner, LoggingMixin):
dag_id=apdr.target_dag_id,
carried_partition_date=apdr.partition_date,
)
- dag_run = dag.create_dagrun(
- run_id=DagRun.generate_run_id(
- run_type=DagRunType.ASSET_TRIGGERED, logical_date=None,
run_after=run_after
- ),
- logical_date=None,
- data_interval=None,
- partition_key=apdr.partition_key,
- partition_date=partition_date,
- run_after=run_after,
- run_type=DagRunType.ASSET_TRIGGERED,
- triggered_by=DagRunTriggeredByType.ASSET,
- state=DagRunState.QUEUED,
- creating_job_id=self.job.id,
- session=session,
- )
+ try:
+ dag_run = dag.create_dagrun(
+ run_id=DagRun.generate_run_id(
+ run_type=DagRunType.ASSET_TRIGGERED,
logical_date=None, run_after=run_after
+ ),
+ logical_date=None,
+ data_interval=None,
+ partition_key=apdr.partition_key,
+ partition_date=partition_date,
+ run_after=run_after,
+ run_type=DagRunType.ASSET_TRIGGERED,
+ triggered_by=DagRunTriggeredByType.ASSET,
+ state=DagRunState.QUEUED,
+ creating_job_id=self.job.id,
+ session=session,
+ )
+ except ParamValidationError:
+ # A stored Dag can hold a default that became invalid after an
upgrade, until re-parsed.
+ self.log.exception("Failed creating DagRun",
dag_id=apdr.target_dag_id)
+ continue
asset_events = session.scalars(
select(AssetEvent).where(
PartitionedAssetKeyLog.asset_partition_dag_run_id ==
apdr.id,
@@ -3162,29 +3168,34 @@ class SchedulerJobRunner(BaseJobRunner, LoggingMixin):
if self._multi_team
else None
)
- for run_after, events in
dag.timetable.group_asset_events(asset_events, triggered_date):
- dag_run = dag.create_dagrun(
- run_id=DagRun.generate_run_id(
- run_type=DagRunType.ASSET_TRIGGERED,
logical_date=None, run_after=run_after
- ),
- logical_date=None,
- data_interval=None,
- run_after=run_after,
- run_type=DagRunType.ASSET_TRIGGERED,
- triggered_by=DagRunTriggeredByType.ASSET,
- state=DagRunState.QUEUED,
- creating_job_id=self.job.id,
- session=session,
- )
- remaining_runs -= 1
- stats.incr("asset.triggered_dagruns",
tags=prune_dict({"team_name": team_name}))
- dag_run.consumed_asset_events.extend(events)
- self.log.info(
- "Created asset-triggered DagRun for '%s': run_id=%s,
consumed %d asset events",
- dag.dag_id,
- dag_run.run_id,
- len(events),
- )
+ try:
+ for run_after, events in
dag.timetable.group_asset_events(asset_events, triggered_date):
+ dag_run = dag.create_dagrun(
+ run_id=DagRun.generate_run_id(
+ run_type=DagRunType.ASSET_TRIGGERED,
logical_date=None, run_after=run_after
+ ),
+ logical_date=None,
+ data_interval=None,
+ run_after=run_after,
+ run_type=DagRunType.ASSET_TRIGGERED,
+ triggered_by=DagRunTriggeredByType.ASSET,
+ state=DagRunState.QUEUED,
+ creating_job_id=self.job.id,
+ session=session,
+ )
+ remaining_runs -= 1
+ stats.incr("asset.triggered_dagruns",
tags=prune_dict({"team_name": team_name}))
+ dag_run.consumed_asset_events.extend(events)
+ self.log.info(
+ "Created asset-triggered DagRun for '%s': run_id=%s,
consumed %d asset events",
+ dag.dag_id,
+ dag_run.run_id,
+ len(events),
+ )
+ except ParamValidationError:
+ # ADRQ rows stay queued so the run is created once the Dag is
fixed.
+ self.log.exception("Failed creating DagRun", dag_id=dag.dag_id)
+ continue
session.flush() # Make consumed events visible to ADRQ cleanup.
# Dequeue rows for events just consumed above and any stale ones.
self._delete_consumed_asset_records(dag,
locked_records=queued_adrqs, session=session)
diff --git a/airflow-core/src/airflow/models/backfill.py
b/airflow-core/src/airflow/models/backfill.py
index e25b18866d2..1f64dfa75b9 100644
--- a/airflow-core/src/airflow/models/backfill.py
+++ b/airflow-core/src/airflow/models/backfill.py
@@ -360,11 +360,10 @@ def _validate_backfill_params(
"Dag has tasks for which depends_on_past=True. "
"You must set reprocess behavior to reprocess completed or
reprocess failed."
)
- if dag_run_conf is not None:
- try:
- dag.params.deep_merge(dag_run_conf).validate()
- except ValueError as e:
- raise InvalidBackfillConf(str(e)) from e
+ try:
+ dag.params.deep_merge(dag_run_conf).validate()
+ except ValueError as e:
+ raise InvalidBackfillConf(str(e)) from e
def _do_dry_run(
diff --git a/airflow-core/src/airflow/serialization/definitions/param.py
b/airflow-core/src/airflow/serialization/definitions/param.py
index 5df0af8596f..0b1f28532e3 100644
--- a/airflow-core/src/airflow/serialization/definitions/param.py
+++ b/airflow-core/src/airflow/serialization/definitions/param.py
@@ -22,6 +22,7 @@ import collections.abc
import copy
from typing import TYPE_CHECKING, Any, Literal
+from airflow._shared.timezones.rfc3339 import is_iso8601_datetime,
is_rfc3339_datetime
from airflow.exceptions import ParamValidationError
from airflow.serialization.definitions.notset import NOTSET, is_arg_set
@@ -29,6 +30,10 @@ if TYPE_CHECKING:
from collections.abc import Iterator, Mapping
+def _check_date_time(value: Any) -> bool:
+ return is_rfc3339_datetime(value) or is_iso8601_datetime(value)
+
+
class SerializedParam:
"""Server-side param class for deserialization."""
@@ -58,13 +63,15 @@ class SerializedParam:
"""
from jsonschema import FormatChecker, validate
+ format_checker = FormatChecker()
+ format_checker.checks("date-time")(_check_date_time)
try:
if not is_arg_set(value := self.value):
raise ValueError("No value passed")
validate(
value,
self.schema,
- format_checker=FormatChecker(),
+ format_checker=format_checker,
)
except Exception:
if not raises:
diff --git a/airflow-core/tests/unit/jobs/test_scheduler_job.py
b/airflow-core/tests/unit/jobs/test_scheduler_job.py
index f5469183f9f..64871b86617 100644
--- a/airflow-core/tests/unit/jobs/test_scheduler_job.py
+++ b/airflow-core/tests/unit/jobs/test_scheduler_job.py
@@ -144,6 +144,7 @@ from airflow.sdk import (
HourWindow,
IdentityMapper,
MinimumCount,
+ Param,
RollupMapper,
SegmentWindow,
StartOfDayMapper,
@@ -14985,6 +14986,45 @@ def test_asset_runs_share_creation_budget(dag_maker,
session, batch_first_dag):
assert session.scalar(select(func.count()).select_from(AssetDagRunQueue))
== (2 if batch_first_dag else 4)
[email protected]_serialized_dag
+def
test_asset_triggered_dag_with_invalid_params_does_not_block_others(dag_maker:
DagMaker, session: Session):
+ asset = Asset(name="invalid-params-asset")
+ dag_ids = ["invalid-params-consumer", "healthy-consumer"]
+ for dag_id in dag_ids:
+ with dag_maker(dag_id=dag_id, schedule=asset, session=session):
+ EmptyOperator(task_id="task")
+ if dag_id == "invalid-params-consumer":
+ # Bypass the parse-time check: simulates a Dag serialized
before the stricter validation.
+ dag_maker.dag.params["d"] = Param("garbage", type="string",
format="date-time")
+ session.commit()
+ asset_id = session.scalar(select(AssetModel.id).where(AssetModel.uri ==
asset.uri))
+ event = AssetEvent(asset_id=asset_id, timestamp=timezone.utcnow() +
timedelta(seconds=1))
+ session.add(event)
+ session.flush()
+ session.add_all(
+ AssetDagRunQueue(asset_id=asset_id, target_dag_id=dag_id,
asset_event_id=event.id)
+ for dag_id in dag_ids
+ )
+ session.flush()
+ dag_models =
session.scalars(select(DagModel).where(DagModel.dag_id.in_(dag_ids))).all()
+
+ runner = SchedulerJobRunner(
+ job=Job(job_type=SchedulerJobRunner.job_type),
executors=[MockExecutor(do_update=False)]
+ )
+ runner._create_dag_runs_asset_triggered(dag_models=dag_models,
session=session)
+ session.flush()
+
+ def count_runs(dag_id):
+ return
session.scalar(select(func.count()).select_from(DagRun).where(DagRun.dag_id ==
dag_id))
+
+ assert count_runs("healthy-consumer") == 1
+ assert count_runs("invalid-params-consumer") == 0
+ queued = session.scalars(
+
select(AssetDagRunQueue.target_dag_id).where(AssetDagRunQueue.target_dag_id.in_(dag_ids))
+ ).all()
+ assert queued == ["invalid-params-consumer"]
+
+
@pytest.mark.need_serialized_dag
@pytest.mark.usefixtures("clear_asset_partition_rows")
@pytest.mark.parametrize(
@@ -15093,6 +15133,45 @@ def
test_partitioned_asset_dag_run_is_not_created_after_a_drain_completes(
assert apdr.created_dag_run_id is None
[email protected]_serialized_dag
[email protected]("clear_asset_partition_rows")
+def test_partitioned_dag_with_invalid_params_keeps_its_apdr_pending(dag_maker:
DagMaker, session: Session):
+ asset = Asset(name="partitioned-invalid-params-asset")
+ bad_dag_id = "partitioned-invalid-params-consumer"
+ for dag_id in (bad_dag_id, "partitioned-healthy-consumer"):
+ with dag_maker(dag_id=dag_id,
schedule=PartitionedAssetTimetable(assets=asset), session=session):
+ EmptyOperator(task_id="task")
+ if dag_id == bad_dag_id:
+ dag_maker.dag.params["d"] = Param("garbage", type="string",
format="date-time")
+ session.commit()
+ _produce_and_register_asset_event(
+ dag_id="partitioned-invalid-params-producer",
+ asset=asset,
+ partition_key="key-1",
+ session=session,
+ dag_maker=dag_maker,
+ )
+
+ runner = SchedulerJobRunner(
+ job=Job(job_type=SchedulerJobRunner.job_type),
executors=[MockExecutor(do_update=False)]
+ )
+ assert runner._create_dagruns_for_partitioned_asset_dags(session=session)
== {
+ bad_dag_id,
+ "partitioned-healthy-consumer",
+ }
+
+ def count_runs(dag_id):
+ return
session.scalar(select(func.count()).select_from(DagRun).where(DagRun.dag_id ==
dag_id))
+
+ assert count_runs("partitioned-healthy-consumer") == 1
+ assert count_runs(bad_dag_id) == 0
+ bad_apdr = session.scalar(
+ select(AssetPartitionDagRun).where(AssetPartitionDagRun.target_dag_id
== bad_dag_id)
+ )
+ assert bad_apdr is not None
+ assert bad_apdr.created_dag_run_id is None
+
+
@pytest.mark.need_serialized_dag
@pytest.mark.usefixtures("clear_asset_partition_rows")
def test_partitioned_dag_run_with_invalid_mapping(dag_maker: DagMaker,
session: Session):
diff --git a/airflow-core/tests/unit/models/test_backfill.py
b/airflow-core/tests/unit/models/test_backfill.py
index 70cc5ca79f7..8c72260a244 100644
--- a/airflow-core/tests/unit/models/test_backfill.py
+++ b/airflow-core/tests/unit/models/test_backfill.py
@@ -50,7 +50,7 @@ from airflow.models.backfill import (
_handle_clear_run,
)
from airflow.providers.standard.operators.python import PythonOperator
-from airflow.sdk import Asset, CronPartitionTimetable,
PartitionedAssetTimetable
+from airflow.sdk import Asset, CronPartitionTimetable, Param,
PartitionedAssetTimetable
from airflow.ti_deps.dep_context import DepContext
from airflow.timetables.base import DagRunInfo, DataInterval
from airflow.utils.state import DagRunState, DagSchedulingState,
TaskInstanceState
@@ -736,7 +736,6 @@ def
test_backfill_empty_conf_overrides_existing_dag_run(dag_maker, session):
def test_backfill_rejects_invalid_conf(dag_maker, session):
"""Backfill with invalid conf should fail validation before creating any
runs."""
- from airflow.sdk import Param
with dag_maker(
schedule="@daily",
@@ -759,9 +758,30 @@ def test_backfill_rejects_invalid_conf(dag_maker, session):
assert session.scalar(select(DagRun).where(DagRun.dag_id == dag.dag_id))
is None
+def test_backfill_without_conf_rejects_invalid_stored_default(dag_maker,
session):
+ """A stored invalid default must fail validation even when no conf is
given, before any row is written."""
+
+ with dag_maker(schedule="@daily") as dag:
+ PythonOperator(task_id="hi", python_callable=print)
+ # Bypass the parse-time check: simulates a Dag serialized before the
stricter validation.
+ dag_maker.dag.params["d"] = Param("garbage", type="string",
format="date-time")
+
+ with pytest.raises(InvalidBackfillConf, match="Invalid input for param d"):
+ _create_backfill(
+ dag_id=dag.dag_id,
+ from_date=pendulum.parse("2021-01-01"),
+ to_date=pendulum.parse("2021-01-05"),
+ max_active_runs=10,
+ reverse=False,
+ triggering_user_name="pytest",
+ dag_run_conf=None,
+ )
+
+ assert session.scalar(select(Backfill).where(Backfill.dag_id ==
dag.dag_id)) is None
+
+
def test_do_dry_run_rejects_invalid_conf(dag_maker, session):
"""Dry run with invalid conf should fail validation."""
- from airflow.sdk import Param
with dag_maker(
schedule="@daily",
diff --git a/airflow-core/tests/unit/serialization/definitions/test_param.py
b/airflow-core/tests/unit/serialization/definitions/test_param.py
index 806417945ea..45027dbf0c1 100644
--- a/airflow-core/tests/unit/serialization/definitions/test_param.py
+++ b/airflow-core/tests/unit/serialization/definitions/test_param.py
@@ -16,7 +16,10 @@
# under the License.
from __future__ import annotations
+from unittest import mock
+
import pytest
+from jsonschema import FormatChecker
from airflow.serialization.definitions.param import SerializedParam
@@ -58,3 +61,16 @@ class TestSerializedParam:
"""Test invalid ISO 8601 duration strings."""
with pytest.raises(Exception, match="is not a 'duration'"):
SerializedParam(duration, type="string",
format="duration").resolve(raises=True)
+
+ @mock.patch.dict(FormatChecker.checkers)
+ def test_string_datetime_validated_without_rfc3339_validator(self):
+ """A default install has no rfc3339-validator, so jsonschema registers
no date-time checker."""
+ FormatChecker.checkers.pop("date-time", None)
+ with pytest.raises(Exception, match="is not a 'date-time'"):
+ SerializedParam("garbage", type="string",
format="date-time").resolve(raises=True)
+ assert SerializedParam("2024-01-01t00:00:00z", type="string",
format="date-time").resolve()
+
+ @pytest.mark.parametrize("value", ["2024-01-01T00:00:00", "2024-01-01
00:00:00+00:00"])
+ def test_string_iso8601_datetime_accepted_unchanged(self, value, recwarn):
+ assert SerializedParam(value, type="string",
format="date-time").resolve(raises=True) == value
+ assert not recwarn.list
diff --git a/shared/timezones/src/airflow_shared/timezones/rfc3339.py
b/shared/timezones/src/airflow_shared/timezones/rfc3339.py
new file mode 100644
index 00000000000..069ae9c5d6a
--- /dev/null
+++ b/shared/timezones/src/airflow_shared/timezones/rfc3339.py
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import re
+from datetime import date, datetime
+
+from pendulum.parsing import parse_iso8601
+
+_RFC3339_DATETIME = re.compile(
+
r"\d{4}-(?:0[1-9]|1[0-2])-\d{2}T(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:\.\d+)?"
+ r"(?:Z|[+-](?:[01]\d|2[0-3]):[0-5]\d)",
+ re.ASCII,
+)
+
+
+# jsonschema checks "date-time" only when the unmaintained
``rfc3339-validator`` is installed.
+def is_rfc3339_datetime(value: object) -> bool:
+ """
+ Return whether ``value`` is an RFC 3339 date-time string, as a JSON Schema
format checker.
+
+ The offset is required, ``T`` and ``Z`` are case-insensitive, and
fractional seconds are allowed.
+ Leap seconds (``:60``) are rejected because ``datetime`` cannot represent
them.
+ Non-strings return ``True``: a format applies only to strings, ``type``
rejects the rest.
+ """
+ if not isinstance(value, str):
+ return True
+ value = value.upper()
+ if _RFC3339_DATETIME.fullmatch(value) is None:
+ return False
+ try:
+ date.fromisoformat(value[:10])
+ except ValueError:
+ return False
+ return True
+
+
+def is_iso8601_datetime(value: str) -> bool:
+ """Return whether ``value`` is an ISO 8601 date-time, the Airflow 2
fallback for non-RFC 3339 values."""
+ try:
+ return isinstance(parse_iso8601(value), datetime)
+ except ValueError:
+ return False
diff --git a/shared/timezones/tests/timezones/test_rfc3339.py
b/shared/timezones/tests/timezones/test_rfc3339.py
new file mode 100644
index 00000000000..dda163fbefa
--- /dev/null
+++ b/shared/timezones/tests/timezones/test_rfc3339.py
@@ -0,0 +1,76 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import pytest
+
+from airflow_shared.timezones.rfc3339 import is_iso8601_datetime,
is_rfc3339_datetime
+
+
[email protected](
+ ("value", "expected"),
+ [
+ ("2024-01-01T00:00:00+00:00", True),
+ ("2024-01-01T00:00:00Z", True),
+ ("2024-01-01t00:00:00z", True),
+ ("2024-01-01T00:00:00.123456Z", True),
+ ("2024-02-29T00:00:00Z", True),
+ ("2024-01-01T00:00:00-23:59", True),
+ ("2024-01-01T00:00:00", False),
+ ("2024-01-01 00:00:00Z", False),
+ ("2024-01-01", False),
+ ("", False),
+ ("garbage", False),
+ ("{{ ds }}", False),
+ ("2024-01-01T00:00Z", False),
+ ("2024-01-01T24:00:00Z", False),
+ ("2024-01-01T00:60:00Z", False),
+ ("2024-01-01T00:00:60Z", False),
+ ("2024-13-01T00:00:00Z", False),
+ ("2023-02-29T00:00:00Z", False),
+ ("0000-01-01T00:00:00Z", False),
+ ("2024-01-01T00:00:00+0000", False),
+ ("2024-01-01T00:00:00+24:00", False),
+ ("2024-01-01T00:00:00+00:60", False),
+ ("2024-01-01T00:00:00Z\n", False),
+ ("2024-01-01T0٠:00:00Z", False),
+ (None, True),
+ (1, True),
+ ([], True),
+ ],
+)
+def test_is_rfc3339_datetime(value, expected):
+ assert is_rfc3339_datetime(value) is expected
+
+
[email protected](
+ ("value", "expected"),
+ [
+ ("2024-01-01T00:00:00+00:00", True),
+ ("2024-01-01T00:00:00", True),
+ ("2024-01-01 00:00:00+00:00", True),
+ ("2024-01-01T00:00Z", True),
+ ("2024-01-01", False),
+ ("03:04:05", False),
+ ("", False),
+ ("garbage", False),
+ ("{{ ds }}", False),
+ ],
+)
+def test_is_iso8601_datetime(value, expected):
+ assert is_iso8601_datetime(value) is expected
diff --git a/task-sdk/src/airflow/sdk/definitions/param.py
b/task-sdk/src/airflow/sdk/definitions/param.py
index ab319d20c9e..b322c477482 100644
--- a/task-sdk/src/airflow/sdk/definitions/param.py
+++ b/task-sdk/src/airflow/sdk/definitions/param.py
@@ -20,12 +20,14 @@ import contextlib
import copy
import json
import logging
+import warnings
from collections.abc import ItemsView, Iterable, Mapping, MutableMapping,
ValuesView
from typing import TYPE_CHECKING, Any, ClassVar, Literal
+from airflow.sdk._shared.timezones.rfc3339 import is_iso8601_datetime,
is_rfc3339_datetime
from airflow.sdk.definitions._internal.mixins import ResolveMixin
from airflow.sdk.definitions._internal.types import NOTSET, is_arg_set
-from airflow.sdk.exceptions import ParamValidationError
+from airflow.sdk.exceptions import ParamValidationError,
RemovedInAirflow4Warning
if TYPE_CHECKING:
from airflow.sdk.definitions.context import Context
@@ -35,6 +37,20 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
+def _check_date_time(value: Any) -> bool:
+ if is_rfc3339_datetime(value):
+ return True
+ if not is_iso8601_datetime(value):
+ return False
+ # stacklevel=1: a deeper level lands in a jsonschema frame, where the
warning is dropped.
+ warnings.warn(
+ f"Param value {value!r} for format='date-time' is deprecated. Use RFC
3339 with a timezone offset.",
+ RemovedInAirflow4Warning,
+ stacklevel=1,
+ )
+ return True
+
+
class Param:
"""
Class to hold the default value of a Param and rule set to do the
validations.
@@ -105,11 +121,13 @@ class Param:
if suppress_exception:
return None
raise ParamValidationError("No value passed and Param has no
default value")
+ format_checker = FormatChecker()
+ format_checker.checks("date-time")(_check_date_time)
try:
validate(
final_val,
self.schema,
- format_checker=FormatChecker(),
+ format_checker=format_checker,
)
except ValidationError as err:
diff --git a/task-sdk/tests/task_sdk/definitions/test_param.py
b/task-sdk/tests/task_sdk/definitions/test_param.py
index 40841129754..e5f443800ef 100644
--- a/task-sdk/tests/task_sdk/definitions/test_param.py
+++ b/task-sdk/tests/task_sdk/definitions/test_param.py
@@ -18,11 +18,13 @@ from __future__ import annotations
from contextlib import nullcontext
from typing import Literal
+from unittest import mock
import pytest
+from jsonschema import FormatChecker
from airflow.sdk.definitions.param import Param, ParamsDict
-from airflow.sdk.exceptions import ParamValidationError
+from airflow.sdk.exceptions import ParamValidationError,
RemovedInAirflow4Warning
from airflow.serialization.definitions.param import SerializedParam
from airflow.serialization.serialized_objects import BaseSerialization
@@ -99,6 +101,19 @@ class TestParam:
with pytest.raises(ParamValidationError, match="is not a 'date-time'"):
Param(dt, type="string", format="date-time").resolve()
+ @mock.patch.dict(FormatChecker.checkers)
+ def test_string_datetime_validated_without_rfc3339_validator(self):
+ """A default install has no rfc3339-validator, so jsonschema registers
no date-time checker."""
+ FormatChecker.checkers.pop("date-time", None)
+ with pytest.raises(ParamValidationError, match="is not a 'date-time'"):
+ Param("garbage", type="string", format="date-time").resolve()
+ assert Param("2024-01-01t00:00:00z", type="string",
format="date-time").resolve()
+
+ @pytest.mark.parametrize("value", ["2024-01-01T00:00:00", "2024-01-01
00:00:00+00:00"])
+ def test_string_iso8601_datetime_warns_and_is_accepted_unchanged(self,
value):
+ with pytest.warns(RemovedInAirflow4Warning, match="is deprecated"):
+ assert Param(value, type="string", format="date-time").resolve()
== value
+
def test_string_time_format(self):
"""Test string time format."""
assert Param("03:04:05", type="string", format="time").resolve() ==
"03:04:05"