Dream95 opened a new pull request, #25773:
URL: https://github.com/apache/pulsar/pull/25773
### Motivation
`reloadBuiltInSources` / `reloadBuiltInSinks` currently reload connectors by
rebuilding the full connector map and closing all existing connector instances.
This causes unnecessary classloader churn and can invalidate previously
referenced connector instances even when the archive content has not changed.
### Modifications
- Added incremental reload logic in `ConnectorUtils.reloadConnectors(...)`:
- Reuse existing `Connector` instances when both archive path and archive
MD5 are unchanged.
- Create new `Connector` instances only for added/changed archives.
- Return evicted connectors (removed/replaced) so callers can close only
those.
- Updated `ConnectorsManager.reloadConnectors(...)` to use the incremental
reload result and close only evicted connectors.
- Extended `Connector` to keep archive MD5 (`archiveMd5Hex`) for identity
checks during reload.
- Extracted MD5 utility methods into `pulsar-common` `FileUtils` and reused
them from both connector reload and `NarUnpacker`.
- Added tests:
- `ConnectorUtilsReloadTest`
- unchanged archive is reused and remains usable
- changed archive is replaced and old connector is closed
- removed archive is evicted and old connector is closed
- `ConnectorsManagerReloadConnectorsTest`
- manager-level reload reuses unchanged connector instance
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- Added unit tests in
`pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/io/ConnectorUtilsReloadTest.java`
- Added runtime-level test in
`pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/worker/ConnectorsManagerReloadConnectorsTest.java`
- CI should cover compilation and execution of the new tests.
### Does this pull request potentially affect one of the following parts:
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Matching PR in forked repository
PR in forked repository: https://github.com/Dream95/pulsar/pull/9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
