ericxiao251 opened a new pull request, #65884:
URL: https://github.com/apache/airflow/pull/65884
# Context
Currently, the worker log groomer sidecar only renders when
`workers.persistence.enabled` is `true` (i.e., StatefulSet mode). This prevents
using the log groomer on Deployments with emptyDir volumes for logs.
This change adds a new `requirePersistence` option that defaults to `true`,
preserving **full backwards compatibility**. When set to `false`, the log
groomer can be enabled regardless of whether persistence is enabled.
This is useful for deployments that:
- Use emptyDir for local logs with remote logging (S3, GCS, etc.)
- Want to clean up local log files to prevent ephemeral storage exhaustion
- Prefer Deployments over StatefulSets for workers
# Configuration
```yaml
workers:
logGroomerSidecar:
enabled: true
requirePersistence: false # Allow log groomer on Deployments
```
# Backward Compatibility
This change is **fully backwards compatible**. The new `requirePersistence`
option defaults to `true`, which preserves the existing behavior where log
groomer only renders when persistence is enabled.
| `persistence` | `enabled` | `requirePersistence` | Log groomer renders |
|---------------|-----------|---------------------|---------------------|
| true | true | true (default) | ✓ |
| true | false | true (default) | ✗ |
| false | true | true (default) | ✗ (existing behavior) |
| false | true | **false** | ✓ (new opt-in behavior) |
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes - Claude (Anthropic)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
