onceMisery opened a new pull request, #25297:
URL: https://github.com/apache/pulsar/pull/25297
- Main Issue: [#25235](https://github.com/apache/pulsar/issues/25235)
### Motivation
This PR improves the Functions status-summary flow across the admin API,
admin client, CLI, and worker runtime.
The original change introduced a namespace-level status-summary endpoint and
related CLI capabilities, but there were still several practical issues:
- newer clients could fail against older workers/brokers that do not expose
the new `/status/summary` endpoint
- newly added admin interface methods could create compatibility pressure
for existing implementations
- the CLI allowed `--state` to be combined with pagination arguments, which
could mislead users because filtering was applied after paging
- worker-side batch status collection was still effectively a per-function
aggregation path and needed safer, controlled parallelism
The goal of this PR is to make the status-summary feature safer to roll out,
clearer to use, and more predictable under load.
### Modifications
This PR includes the following changes:
- adds `default` implementations for the new admin API status-summary
methods in `Functions` to reduce compatibility risk for existing implementations
- aligns the `limit` contract in the admin API documentation with the actual
validation behavior
- list functions first
- query function status per function for the requested page only
- keeps the fallback path page-aware by applying sorting and pagination
before querying individual function statuses
- preserves partial-failure isolation in the fallback path by returning
`UNKNOWN` summaries for failed function-level status lookups
- rejects the misleading CLI combination of `--state` with `--limit` or
`--continuation-token`
- updates CLI option descriptions to clarify the above behavior
- changes worker-side status-summary collection from serial execution to
controlled parallel execution
- adds a new worker configuration item,
`functionsStatusSummaryMaxParallelism`, to bound worker-side batch summary
concurrency
- documents the new worker configuration in `conf/functions_worker.yml`
- adds focused tests for compatibility fallback, pagination correctness, CLI
validation, and configured worker-side parallelism
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- Ran admin client tests:
- `mvn -pl pulsar-client-admin
-Dtest=org.apache.pulsar.client.admin.internal.FunctionsImplTest
-DfailIfNoTests=false test -q`
- Ran CLI tests:
- `mvn -pl pulsar-client-tools-test
-Dtest=org.apache.pulsar.admin.cli.CmdFunctionsTest -DfailIfNoTests=false test
-q`
- Built the runtime module required by the new worker configuration field:
- `mvn -pl pulsar-functions/runtime -DskipTests install -q`
- Ran worker tests:
- `mvn -pl pulsar-functions/worker
-Dtest=org.apache.pulsar.functions.worker.rest.api.FunctionsImplTest
-DfailIfNoTests=false test -q`
- Ran checkstyle for the updated admin client module:
- `mvn -pl pulsar-client-admin -DskipTests
'-Dpulsar.basedir=D:/code/source-code/pulsar' checkstyle:check -q`
### Does this pull request potentially affect one of the following parts:
- [ ] Dependencies (add or upgrade a dependency)
- [x] The public API
- [ ] The schema
- [x] The default values of configurations
- [x] The threading model
- [ ] The binary protocol
- [x] The REST endpoints
- [x] The admin CLI options
- [ ] The metrics
- [x] Anything that affects deployment
Highlights:
- Public API: adds compatible `default` implementations for newly introduced
status-summary methods
- Default values of configurations: adds
`functionsStatusSummaryMaxParallelism` with default value `8`
- Threading model: worker-side batch summary collection now uses bounded
parallel execution instead of serial execution
- REST endpoints: this change improves compatibility handling for the new
status-summary endpoint
- Admin CLI options: validates and restricts `--state` with pagination
options to avoid ambiguous behavior
- Deployment: operators can now tune the max parallelism used by worker-side
status-summary queries
### Documentation
- [ ] `doc`
- [x] `doc-required`
- [ ] `doc-not-needed`
- [ ] `doc-complete`
### Matching PR in forked repository
[PR in forked repository: <!-- ENTER URL HERE
-->](https://github.com/onceMisery/pulsar)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
