Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
vatsrahul1001 merged PR #67233: URL: https://github.com/apache/airflow/pull/67233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
vatsrahul1001 commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3278682809 ## RELEASE_NOTES.rst: ## @@ -24,6 +24,227 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) +- Fix triggerer race condition and deadlock that caused deferred tasks to stall indefinitely + + Triggers that call synchronous SDK methods (e.g. ``get_task_states`` used by + ``safe_to_cancel`` in several Google provider operators) could crash the triggerer's + internal subprocess. The triggerer would then continue to heartbeat normally — + appearing healthy to the scheduler — while silently processing zero triggers, causing + every deferred task to time out. This was first reported in issue #64620; a + partial fix shipped in Airflow 3.2.1 (#64882) but introduced a new deadlock + with the same visible symptom under load. + + Both issues are fixed by replacing the lock-based serialization with response + multiplexing: each request now carries a unique ID and the response is routed back to + the correct caller, so concurrent requests from trigger threads no longer contend or + deadlock regardless of how many triggers are running or what SDK methods they call. + + **New: triggerer subprocess watchdog** + + Even with the race fixed, a trigger that blocks the event loop (e.g. by calling + ``time.sleep()`` or performing blocking I/O directly in ``async def run()``) would + previously leave the triggerer appearing healthy indefinitely. + + A new ``[triggerer] runner_health_check_threshold`` config option (default: 30 seconds) + adds a watchdog: if the triggerer subprocess goes silent for longer than the threshold, + the parent process stops updating the heartbeat so the scheduler can detect the hang and + reassign triggers rather than waiting for them to individually time out. Set the option + to ``0`` to disable the watchdog. (#66412) + + +- Tighten ``[core] allowed_deserialization_classes_regexp`` to require full-string matches + + Patterns in ``[core] allowed_deserialization_classes_regexp`` are now matched + against the entire classname using ``re.fullmatch()`` instead of ``re.match()``. + Previously a pattern such as ``airflow\.models\.Variable`` admitted not only + the intended class but also names that started with it + (e.g. ``airflow.models.Variable_Malicious``), because ``re.match`` only anchors + at the start of the string. + + The default value of this option is empty, so out-of-the-box deployments are + unaffected. Deployments that configured this option with patterns relying on + prefix-match semantics — for example ``airflow\.models\.`` to mean "any class + under ``airflow.models``" — must add ``.*`` to the pattern + (``airflow\.models\..*``) to retain the previous behaviour. (#66499) + +- Custom deadline reference classes must now be registered via the new ``deadline_references`` attribute on ``AirflowPlugin``, matching the existing pattern for custom timetables and custom partition mappers. To use a custom ``DeadlineReference`` subclass, register it in a plugin's ``deadline_references`` list. Custom references that are not registered will raise ``DeadlineReferenceNotRegistered`` at deserialization. (#66737) + +Bug Fixes
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
amoghrajesh commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3278578682 ## RELEASE_NOTES.rst: ## @@ -24,6 +24,227 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) +- Fix triggerer race condition and deadlock that caused deferred tasks to stall indefinitely + + Triggers that call synchronous SDK methods (e.g. ``get_task_states`` used by + ``safe_to_cancel`` in several Google provider operators) could crash the triggerer's + internal subprocess. The triggerer would then continue to heartbeat normally — + appearing healthy to the scheduler — while silently processing zero triggers, causing + every deferred task to time out. This was first reported in issue #64620; a + partial fix shipped in Airflow 3.2.1 (#64882) but introduced a new deadlock + with the same visible symptom under load. + + Both issues are fixed by replacing the lock-based serialization with response + multiplexing: each request now carries a unique ID and the response is routed back to + the correct caller, so concurrent requests from trigger threads no longer contend or + deadlock regardless of how many triggers are running or what SDK methods they call. + + **New: triggerer subprocess watchdog** + + Even with the race fixed, a trigger that blocks the event loop (e.g. by calling + ``time.sleep()`` or performing blocking I/O directly in ``async def run()``) would + previously leave the triggerer appearing healthy indefinitely. + + A new ``[triggerer] runner_health_check_threshold`` config option (default: 30 seconds) + adds a watchdog: if the triggerer subprocess goes silent for longer than the threshold, + the parent process stops updating the heartbeat so the scheduler can detect the hang and + reassign triggers rather than waiting for them to individually time out. Set the option + to ``0`` to disable the watchdog. (#66412) + + +- Tighten ``[core] allowed_deserialization_classes_regexp`` to require full-string matches + + Patterns in ``[core] allowed_deserialization_classes_regexp`` are now matched + against the entire classname using ``re.fullmatch()`` instead of ``re.match()``. + Previously a pattern such as ``airflow\.models\.Variable`` admitted not only + the intended class but also names that started with it + (e.g. ``airflow.models.Variable_Malicious``), because ``re.match`` only anchors + at the start of the string. + + The default value of this option is empty, so out-of-the-box deployments are + unaffected. Deployments that configured this option with patterns relying on + prefix-match semantics — for example ``airflow\.models\.`` to mean "any class + under ``airflow.models``" — must add ``.*`` to the pattern + (``airflow\.models\..*``) to retain the previous behaviour. (#66499) + +- Custom deadline reference classes must now be registered via the new ``deadline_references`` attribute on ``AirflowPlugin``, matching the existing pattern for custom timetables and custom partition mappers. To use a custom ``DeadlineReference`` subclass, register it in a plugin's ``deadline_references`` list. Custom references that are not registered will raise ``DeadlineReferenceNotRegistered`` at deserialization. (#66737) + +Bug Fixes +
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
vatsrahul1001 commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3274977649 ## RELEASE_NOTES.rst: ## @@ -24,6 +24,225 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. (#64207) +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) +- Fix triggerer race condition and deadlock that caused deferred tasks to stall indefinitely + + Triggers that call synchronous SDK methods (e.g. ``get_task_states`` used by + ``safe_to_cancel`` in several Google provider operators) could crash the triggerer's + internal subprocess. The triggerer would then continue to heartbeat normally — + appearing healthy to the scheduler — while silently processing zero triggers, causing + every deferred task to time out. This was first reported in issue #64620; a + partial fix shipped in Airflow 3.2.1 (#64882) but introduced a new deadlock + with the same visible symptom under load. + + Both issues are fixed by replacing the lock-based serialization with response + multiplexing: each request now carries a unique ID and the response is routed back to + the correct caller, so concurrent requests from trigger threads no longer contend or + deadlock regardless of how many triggers are running or what SDK methods they call. + + **New: triggerer subprocess watchdog** + + Even with the race fixed, a trigger that blocks the event loop (e.g. by calling + ``time.sleep()`` or performing blocking I/O directly in ``async def run()``) would + previously leave the triggerer appearing healthy indefinitely. + + A new ``[triggerer] runner_health_check_threshold`` config option (default: 30 seconds) + adds a watchdog: if the triggerer subprocess goes silent for longer than the threshold, + the parent process stops updating the heartbeat so the scheduler can detect the hang and + reassign triggers rather than waiting for them to individually time out. Set the option + to ``0`` to disable the watchdog. (#66412) + + +- Tighten ``[core] allowed_deserialization_classes_regexp`` to require full-string matches Review Comment: Discussed with @potiuk we do not have any CVE for this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
vatsrahul1001 commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3274946651 ## RELEASE_NOTES.rst: ## @@ -24,6 +24,226 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. Dashboards built against the previous Gauge representation must migrate to Histogram aggregations (count/sum/bucket) to keep working — Grafana panels using ``gauge`` queries on Airflow timer metrics will need updating to ``histogram_quantile`` (or equivalent) on the new buckets. (#64207) Review Comment: Yeah I agree. I have removed better suited for 3.3.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
vatsrahul1001 commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3274265334 ## providers/cncf/kubernetes/docs/changelog.rst: ## Review Comment: Yeah -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
pierrejeambrun commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3274122778 ## providers/cncf/kubernetes/docs/changelog.rst: ## Review Comment: not sure why we have provider change in here. I assume it's a side effect of other backports? ## RELEASE_NOTES.rst: ## @@ -24,6 +24,226 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. Dashboards built against the previous Gauge representation must migrate to Histogram aggregations (count/sum/bucket) to keep working — Grafana panels using ``gauge`` queries on Airflow timer metrics will need updating to ``histogram_quantile`` (or equivalent) on the new buckets. (#64207) Review Comment: That doesn't sound like it belongs to a patch release, maybe push this to '3.3.0' ? It's fixing things but at the same time it's a behavioral change breaking people dashboard. ## RELEASE_NOTES.rst: ## @@ -24,6 +24,225 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. (#64207) +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) Review Comment: Yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
pierrejeambrun commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3274085674 ## RELEASE_NOTES.rst: ## @@ -24,6 +24,225 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. (#64207) +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) +- Fix triggerer race condition and deadlock that caused deferred tasks to stall indefinitely + + Triggers that call synchronous SDK methods (e.g. ``get_task_states`` used by + ``safe_to_cancel`` in several Google provider operators) could crash the triggerer's + internal subprocess. The triggerer would then continue to heartbeat normally — + appearing healthy to the scheduler — while silently processing zero triggers, causing + every deferred task to time out. This was first reported in issue #64620; a + partial fix shipped in Airflow 3.2.1 (#64882) but introduced a new deadlock + with the same visible symptom under load. + + Both issues are fixed by replacing the lock-based serialization with response + multiplexing: each request now carries a unique ID and the response is routed back to + the correct caller, so concurrent requests from trigger threads no longer contend or + deadlock regardless of how many triggers are running or what SDK methods they call. + + **New: triggerer subprocess watchdog** + + Even with the race fixed, a trigger that blocks the event loop (e.g. by calling + ``time.sleep()`` or performing blocking I/O directly in ``async def run()``) would + previously leave the triggerer appearing healthy indefinitely. + + A new ``[triggerer] runner_health_check_threshold`` config option (default: 30 seconds) + adds a watchdog: if the triggerer subprocess goes silent for longer than the threshold, + the parent process stops updating the heartbeat so the scheduler can detect the hang and + reassign triggers rather than waiting for them to individually time out. Set the option + to ``0`` to disable the watchdog. (#66412) + + +- Tighten ``[core] allowed_deserialization_classes_regexp`` to require full-string matches + + Patterns in ``[core] allowed_deserialization_classes_regexp`` are now matched + against the entire classname using ``re.fullmatch()`` instead of ``re.match()``. + Previously a pattern such as ``airflow\.models\.Variable`` admitted not only + the intended class but also names that started with it + (e.g. ``airflow.models.Variable_Malicious``), because ``re.match`` only anchors + at the start of the string. + + The default value of this option is empty, so out-of-the-box deployments are + unaffected. Deployments that configured this option with patterns relying on + prefix-match semantics — for example ``airflow\.models\.`` to mean "any class + under ``airflow.models``" — must add ``.*`` to the pattern + (``airflow\.models\..*``) to retain the previous behaviour. (#66499) + +- Custom deadline reference classes must now be registered via the new ``deadline_references`` attribute on ``AirflowPlugin``, matching the existing pattern for custom timetables and custom partition mappers. To use a custom ``DeadlineReference`` subclass, register it in a plugin's ``deadline_ref
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
vatsrahul1001 commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3273570047 ## RELEASE_NOTES.rst: ## @@ -24,6 +24,225 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. (#64207) +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) +- Fix triggerer race condition and deadlock that caused deferred tasks to stall indefinitely + + Triggers that call synchronous SDK methods (e.g. ``get_task_states`` used by + ``safe_to_cancel`` in several Google provider operators) could crash the triggerer's + internal subprocess. The triggerer would then continue to heartbeat normally — + appearing healthy to the scheduler — while silently processing zero triggers, causing + every deferred task to time out. This was first reported in issue #64620; a + partial fix shipped in Airflow 3.2.1 (#64882) but introduced a new deadlock + with the same visible symptom under load. + + Both issues are fixed by replacing the lock-based serialization with response + multiplexing: each request now carries a unique ID and the response is routed back to + the correct caller, so concurrent requests from trigger threads no longer contend or + deadlock regardless of how many triggers are running or what SDK methods they call. + + **New: triggerer subprocess watchdog** + + Even with the race fixed, a trigger that blocks the event loop (e.g. by calling + ``time.sleep()`` or performing blocking I/O directly in ``async def run()``) would + previously leave the triggerer appearing healthy indefinitely. + + A new ``[triggerer] runner_health_check_threshold`` config option (default: 30 seconds) + adds a watchdog: if the triggerer subprocess goes silent for longer than the threshold, + the parent process stops updating the heartbeat so the scheduler can detect the hang and + reassign triggers rather than waiting for them to individually time out. Set the option + to ``0`` to disable the watchdog. (#66412) + + +- Tighten ``[core] allowed_deserialization_classes_regexp`` to require full-string matches + + Patterns in ``[core] allowed_deserialization_classes_regexp`` are now matched + against the entire classname using ``re.fullmatch()`` instead of ``re.match()``. + Previously a pattern such as ``airflow\.models\.Variable`` admitted not only + the intended class but also names that started with it + (e.g. ``airflow.models.Variable_Malicious``), because ``re.match`` only anchors + at the start of the string. + + The default value of this option is empty, so out-of-the-box deployments are + unaffected. Deployments that configured this option with patterns relying on + prefix-match semantics — for example ``airflow\.models\.`` to mean "any class + under ``airflow.models``" — must add ``.*`` to the pattern + (``airflow\.models\..*``) to retain the previous behaviour. (#66499) + +- Custom deadline reference classes must now be registered via the new ``deadline_references`` attribute on ``AirflowPlugin``, matching the existing pattern for custom timetables and custom partition mappers. To use a custom ``DeadlineReference`` subclass, register it in a plugin's ``deadline_refe
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
vatsrahul1001 commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3273389224 ## RELEASE_NOTES.rst: ## @@ -24,6 +24,225 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. (#64207) +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) +- Fix triggerer race condition and deadlock that caused deferred tasks to stall indefinitely + + Triggers that call synchronous SDK methods (e.g. ``get_task_states`` used by + ``safe_to_cancel`` in several Google provider operators) could crash the triggerer's + internal subprocess. The triggerer would then continue to heartbeat normally — + appearing healthy to the scheduler — while silently processing zero triggers, causing + every deferred task to time out. This was first reported in issue #64620; a + partial fix shipped in Airflow 3.2.1 (#64882) but introduced a new deadlock + with the same visible symptom under load. + + Both issues are fixed by replacing the lock-based serialization with response + multiplexing: each request now carries a unique ID and the response is routed back to + the correct caller, so concurrent requests from trigger threads no longer contend or + deadlock regardless of how many triggers are running or what SDK methods they call. + + **New: triggerer subprocess watchdog** + + Even with the race fixed, a trigger that blocks the event loop (e.g. by calling + ``time.sleep()`` or performing blocking I/O directly in ``async def run()``) would + previously leave the triggerer appearing healthy indefinitely. + + A new ``[triggerer] runner_health_check_threshold`` config option (default: 30 seconds) + adds a watchdog: if the triggerer subprocess goes silent for longer than the threshold, + the parent process stops updating the heartbeat so the scheduler can detect the hang and + reassign triggers rather than waiting for them to individually time out. Set the option + to ``0`` to disable the watchdog. (#66412) + + +- Tighten ``[core] allowed_deserialization_classes_regexp`` to require full-string matches + + Patterns in ``[core] allowed_deserialization_classes_regexp`` are now matched + against the entire classname using ``re.fullmatch()`` instead of ``re.match()``. + Previously a pattern such as ``airflow\.models\.Variable`` admitted not only + the intended class but also names that started with it + (e.g. ``airflow.models.Variable_Malicious``), because ``re.match`` only anchors + at the start of the string. + + The default value of this option is empty, so out-of-the-box deployments are + unaffected. Deployments that configured this option with patterns relying on + prefix-match semantics — for example ``airflow\.models\.`` to mean "any class + under ``airflow.models``" — must add ``.*`` to the pattern + (``airflow\.models\..*``) to retain the previous behaviour. (#66499) + +- Custom deadline reference classes must now be registered via the new ``deadline_references`` attribute on ``AirflowPlugin``, matching the existing pattern for custom timetables and custom partition mappers. To use a custom ``DeadlineReference`` subclass, register it in a plugin's ``deadline_refe
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
vatsrahul1001 commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3273365533 ## RELEASE_NOTES.rst: ## @@ -24,6 +24,225 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. (#64207) +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) +- Fix triggerer race condition and deadlock that caused deferred tasks to stall indefinitely + + Triggers that call synchronous SDK methods (e.g. ``get_task_states`` used by + ``safe_to_cancel`` in several Google provider operators) could crash the triggerer's + internal subprocess. The triggerer would then continue to heartbeat normally — + appearing healthy to the scheduler — while silently processing zero triggers, causing + every deferred task to time out. This was first reported in issue #64620; a + partial fix shipped in Airflow 3.2.1 (#64882) but introduced a new deadlock + with the same visible symptom under load. + + Both issues are fixed by replacing the lock-based serialization with response + multiplexing: each request now carries a unique ID and the response is routed back to + the correct caller, so concurrent requests from trigger threads no longer contend or + deadlock regardless of how many triggers are running or what SDK methods they call. + + **New: triggerer subprocess watchdog** + + Even with the race fixed, a trigger that blocks the event loop (e.g. by calling + ``time.sleep()`` or performing blocking I/O directly in ``async def run()``) would + previously leave the triggerer appearing healthy indefinitely. + + A new ``[triggerer] runner_health_check_threshold`` config option (default: 30 seconds) + adds a watchdog: if the triggerer subprocess goes silent for longer than the threshold, + the parent process stops updating the heartbeat so the scheduler can detect the hang and + reassign triggers rather than waiting for them to individually time out. Set the option + to ``0`` to disable the watchdog. (#66412) + + +- Tighten ``[core] allowed_deserialization_classes_regexp`` to require full-string matches Review Comment: As per PR it was resolution of issue found in ASVS scan of `airflow/task-sdk`. @potiuk do we have CVE/GHSA reference for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
vatsrahul1001 commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3273345449 ## RELEASE_NOTES.rst: ## @@ -24,6 +24,225 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. (#64207) +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) +- Fix triggerer race condition and deadlock that caused deferred tasks to stall indefinitely + + Triggers that call synchronous SDK methods (e.g. ``get_task_states`` used by + ``safe_to_cancel`` in several Google provider operators) could crash the triggerer's + internal subprocess. The triggerer would then continue to heartbeat normally — + appearing healthy to the scheduler — while silently processing zero triggers, causing + every deferred task to time out. This was first reported in issue #64620; a + partial fix shipped in Airflow 3.2.1 (#64882) but introduced a new deadlock + with the same visible symptom under load. + + Both issues are fixed by replacing the lock-based serialization with response + multiplexing: each request now carries a unique ID and the response is routed back to + the correct caller, so concurrent requests from trigger threads no longer contend or + deadlock regardless of how many triggers are running or what SDK methods they call. + + **New: triggerer subprocess watchdog** + + Even with the race fixed, a trigger that blocks the event loop (e.g. by calling + ``time.sleep()`` or performing blocking I/O directly in ``async def run()``) would + previously leave the triggerer appearing healthy indefinitely. + + A new ``[triggerer] runner_health_check_threshold`` config option (default: 30 seconds) Review Comment: Yeah I will add in upgrade docs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
vatsrahul1001 commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3273343352 ## RELEASE_NOTES.rst: ## @@ -24,6 +24,225 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. (#64207) +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) Review Comment: Yeah I did confirm with @pierrejeambrun and check myself as well "Match anywhere" toggle is wired up on every search -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Sync v3-2-stable with v3-2-test to release 3.2.2rc1 [airflow]
kaxil commented on code in PR #67233: URL: https://github.com/apache/airflow/pull/67233#discussion_r3273155211 ## RELEASE_NOTES.rst: ## @@ -24,6 +24,225 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. (#64207) +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) Review Comment: UI exception applies and the opt-back-in toggle ships, so this is fine for a patch, but worth confirming before tagging rc1 that the "Match anywhere" toggle is actually wired up on every search input and filter pill the bullet claims. ## RELEASE_NOTES.rst: ## @@ -24,6 +24,225 @@ .. towncrier release notes start +Airflow 3.2.2 (2026-05-26) +-- + +Significant Changes +^^^ + +- OTel timer and timing metrics now use Histogram instead of Gauge, preserving count, sum, and bucket distribution across recordings. (#64207) +- The SMTP STARTTLS upgrade performed by ``airflow.utils.email.send_email`` now validates the SMTP server's certificate against the system's trusted CA bundle by default. Previously the ``starttls()`` call was made without an SSL context, so any certificate was accepted. + Deployments that intentionally point Airflow at an SMTP server with a self-signed or otherwise non-validating certificate and need to preserve the previous behaviour must set ``email.ssl_context = "none"`` in ``airflow.cfg``. The ``"default"`` value (now also the default when the option is unset) uses :func:`ssl.create_default_context`. Previously this option applied only to the ``SMTP_SSL`` path; it now applies to the STARTTLS path as well. (#65346) +- In #64963, the Airflow UI switched from full-match ``*_pattern`` REST API query parameters to the new index-friendly ``*_prefix_pattern`` parameters on list endpoints. This is a behavioral change for search-as-you-type filters in the UI: matches are prefix-based (``LIKE 'term%'`` via a range scan) instead of substring-based (``ILIKE '%term%'``), which means the database can use B-tree indexes and search stays fast on large deployments. The REST API itself keeps both forms: existing ``*_pattern`` parameters still behave exactly as before. + In #66015, a per-search-bar "Match anywhere" toggle was added so users who relied on the previous substring behavior can opt back into it from the UI. Each search input and each text filter pill now has a small regex-icon toggle next to the value; flipping it on switches that input from ``*_prefix_pattern`` to ``*_pattern``. (#66015) +- Fix triggerer race condition and deadlock that caused deferred tasks to stall indefinitely + + Triggers that call synchronous SDK methods (e.g. ``get_task_states`` used by + ``safe_to_cancel`` in several Google provider operators) could crash the triggerer's + internal subprocess. The triggerer would then continue to heartbeat normally — + appearing healthy to the scheduler — while silently processing zero triggers, causing + every deferred task to time out. This was first reported in issue #64620; a + partial fix shipped in Airflow 3.2.1 (#64882) but introduced a new deadlock + with the same visible symptom under load. + + Both issues a
