CalvinKirs opened a new pull request, #70358:
URL: https://github.com/apache/doris/pull/70358
### Background
The Doris Ranger controller checks ADMIN or SELECT through the resource
hierarchy before checking individual columns. Large direct-user policies can
contain hundreds of resource alternatives each, retaining repeated descriptors
and repeatedly scanning unrelated blocks during authorization.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Performance
Problem Summary:
Reduce retained policy memory and local read-authorization work for large
direct-user policy sets. Share equal resource descriptors within a downloaded
payload, omit parent checks that cannot grant access, and index eligible column
checks by table and user while reusing installed native Ranger matchers.
Unsupported policy semantics, auditing and unsafe matcher cases continue
through native evaluation.
### Details
- **Change map:** 6 production files (+1016/-1 lines), 5 test files (+2323),
5 standalone benchmark source files (+965), and 5 benchmark/applicability
documents (+800). No generated files.
- **Why it was slow:** Per-user policies repeat catalog/database/table
descriptors across many alternatives. Broad candidate selection and scans of
unrelated alternatives are repeated for unsuccessful parent checks and column
actions.
- **Optimization:** Use bounded, payload-local descriptor sharing while
preserving native HashMap behavior. Bind a conservative parent-scope summary
and table/user resource index to the installed engine, policy version and
configuration. Column entries store user IDs, action flags and references to
existing native matchers; no query-result cache or column-set copy is added.
Full replacements and verified copy-on-update deltas rebuild the index;
in-place updates disable it. Cleanup releases snapshot references without
holding the install monitor while joining the native refresher.
- **Results:** Historical measurements on a shared host used 5000 physical
tables, 800 resource blocks per user, three real columns plus a user-named
column, 20 warmups and 120 mixed allow/deny samples. At 3000 users, descriptor
sharing reduced retained policy heap by about 14.45% in the first-stage
comparison. A later index comparison measured first-stage versus indexed
authorization P50 at 676.283 versus 0.069 ms, with about 25.10 MiB additional
retained heap. These index measurements are from `a536d851c95`, before the
final Unicode and dynamic-matcher correctness guards; they are not a benchmark
of the final HEAD. All measurements cover isolated authorization, not
end-to-end SQL or production concurrency. The checked-in reports retain
source/version provenance and reproduction commands.
- **Applicability and cost:** The fast path requires eligible static
direct-user ADMIN/SELECT policies. Deny/exception, group/role,
condition/schedule, tag/zone, custom matcher and other unsupported cases fall
back. Registered result processors preserve native calls and audit callbacks.
Users whose installed resource matchers require dynamic evaluation or whose
resource values contain non-ASCII text fall back to native column checks; this
includes some user-named fake columns. Non-ASCII or ambiguous request prefixes
and non-ASCII columns also fall back. The resource index is bounded at four
million bindings. Native policy memory remains, and installation/refresh are
not guaranteed to improve: the historical 3000-user comparison observed 35.5
versus 43.8 seconds for initial installation and 8134 versus 9207 MiB for
sampled refresh heap peaks. These overall observations do not isolate index
construction time or establish a safe FE heap bound.
### Testing
On final HEAD `6f6fcbc64721cb5b6bfcc4bd1446f2f837d044fa`, local validation
passed **208 tests with one existing Hive test skipped**, with the Maven build
cache disabled:
- Module package: **157 passed / 1 existing skip**, including 16 new
correctness methods.
- Five FE integration gates: **51 passed**. Unchanged FE main classes were
reused (`maven.main.skip=true`); the current plugin classes were used.
- Independent embedded native Ranger engines and real Doris controller
checks compare 6400 randomized complete authorization decisions, plus fresh
per-action requests. Tests require active indexing and unaffected-user
zero-native-call paths, so disabling every fast path cannot satisfy them.
- Regressions cover full/COW-delta revocation, deterministic
install/publication gaps, close/refresh overlap, in-place native revocation,
rejected same-version installs, binding-limit fallback, delegate-admin pruning,
Unicode candidate-selection parity and escaped-resource expansion at all four
resource levels. Both Unicode and paired-backslash overgrants were reproduced
before their fixes and pass afterward.
- Audit tests compare ordered native result sequences and processor removal.
Authorization behavior/API golden files were not regenerated. The native Ranger
2.9.0 artifact is unchanged; packaged plugin classes match the tested source.
```bash
mvn -f fe/pom.xml \
-pl
:fe-authorization-plugin-ranger-doris,:fe-authorization-plugin-ranger-hive -am
package \
-Dmaven.build.cache.enabled=false -DfailIfNoTests=false
mvn -f fe/pom.xml -pl :fe-core -am test \
-Dmaven.build.cache.enabled=false -Dmaven.main.skip=true \
-Dmaven.compiler.useIncrementalCompilation=false -DfailIfNoTests=false \
-Dtest=AccessControllerManagerTest,AccessTranslationTest,LegacyAccessControllerPluginTest,AccessControlBehaviorBaselineTest,AuthorizationPluginFromDirectoryTest
```
The installed-plugin FE gate uses the example authorization plugin. No live
Ranger Admin/FE SQL deployment, AD/OIDC login, real audit sink, concurrent
capacity test or fresh performance run of the final guards has been completed.
Earlier Ranger 2.8 library smoke results are historical compatibility evidence,
not an older-FE binary deployment test.
### Notes for Reviewers
The implementation uses version-coupled public getters from the pinned
Ranger library, isolated in `RangerReadResourceIndex`; a different engine
implementation falls back. Native trie candidate selection and final matching
can differ for Unicode case folding, and exact ASCII resources can still
require request context for escape expansion. The final applicability checks
deliberately preserve native decisions instead of reproducing those rules.
### Release note
Reduce retained Ranger resource memory and accelerate eligible direct-user
read authorization. Complex policies, audited checks and resources requiring
native Unicode or dynamic matching retain native evaluation.
### Check List (For Author)
- Behavior changed: No intended authorization decision change; unsupported
cases use native evaluation.
- Does this need documentation: Yes; in-tree applicability, configuration
and benchmark documentation is included. Operator-facing website notes can
follow validation of the final performance scope.
- Pre-submission status: Partially verified: final performance remeasurement
and live Ranger Admin/FE deployment remain outstanding; keep this PR in draft.
### AI Disclosure
- Tool: Codex (GPT-6)
- Generated: Implementation, tests, benchmark tooling, documentation and
this PR description
- Human review: Pending maintainer review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
