morningman commented on PR #68220: URL: https://github.com/apache/doris/pull/68220#issuecomment-5749739991
@CalvinKirs You're right, and it was worse than one download: `RangerBasePlugin.init()` runs `PolicyRefresher.startRefresher()` on the calling thread, which loads the roles and the policies before it returns, and with the enricher on the policies the user store enricher's `init()` does its first full download inline as well - all of it in the `Env` constructor for ranger-doris. So the user store was a third REST call on a startup path that already had two, each bounded only by the plugin's REST timeouts and retries (it is also the pattern Ranger's own `use.rangerGroups` follows in HiveServer2 and the NameNode, but that is no reason for an FE to wait). Rather than deferring only the user store, 6669fb67462 moves the whole first load off the calling thread: `BackgroundLoadedRangerPlugin` (ranger-common, now the base of both plugins) runs `RangerBasePlugin.init()` on its own `RangerPluginLoader` thread and the constructor returns at once. The first authorization check waits for that load to end (`awaitLoaded()`: policies from the admin or the local cache, user store likewise), so nothing is ever answered out of an empty engine, and the groups of a request are read after the load too - otherwise a group deny could be bypassed by a request built before the store arrived. `cleanup()` during the load returns immediately and the loader stops the plugin once it has finished. Measured locally: against a healthy admin the controller is built 1 ms after the plugin and the load takes 861 ms in the background, before the FE is master. Against an unreachable admin the FE is master in 7 s and opens its query port at 40 s while the load runs for 70 s; a `SHOW DATABASES` issued at 40 s by a user in a Ranger group waits 36 s and then answers out of the cached policies and user store, while `SELECT 1` answers at once. One related thing worth knowing, not about startup: with `supports.policy.deltas` off (Ranger's default), every policy version change rebuilds the engine and its enrichers, so the full user store is downloaded again on each policy edit, on the refresher thread. That is how Ranger's implicit enricher behaves as well; `ranger.plugin.doris.supports.policy.deltas=true` shares the enricher across delta updates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
