smolnar82 opened a new pull request, #826: URL: https://github.com/apache/knox/pull/826
## What changes were proposed in this pull request? Implemented the changes I listed in [KNOX-2990](https://issues.apache.org/jira/browse/KNOX-2990): - deprecated the following TSS implementations: - AliasBasedTokenStateService - ZookeeperTokenStateService - JournalBasedTokenStateService - implemented a DerbyDB storage that stores tokens in `$DATA_DIR/security/tokens` (this time it's not yet encrypted) - file permissions are set on that folder to `700` (only the owner can access it) - changed the default implementation in `TokenStateServiceFactory` to the new DerbyDatabaseTSS - implemenedt a new KnoxCLI command that migrates existing tokens from credential stores to any JDBC-based TSS backend (tested it with the new DerbyDB TSS; see below) - integrated this new KnoxCLI command in a way such that it runs when Knox Gateway is started: if token management is enabled, and the configured TSS implementation is a migration target (currently it's true for any JDBC-based TSS implementation) - added some new `gateway-site.xml` properties: - `gateway.knox.token.migration.skip `: ensures that the previous automated step can be controlled (E.g. in case of unforeseen errors it can be turned off). Defaults - `gateway.knox.token.migration.archive.tokens `: indicates if migrated tokesn should be archived in another credential store called `__tokens`. Defaults to `false`. - `gateway.knox.token.migration.include.expired.tokens`: whether expired tokens should be migrated or skipped. Defaults to `false`. - `gateway.knox.token.migration.verbose`: if true, migrated/skipped tokens are added in the `[gateway|knoxcli].log` and, optionally, on the STDOUT (when running the KnoxCLI tool manually). Defaults to `true`, because it's very useful to have the chancee to cross-reference token IDs in case of error debugging. - `gateway.knox.token.migration.progress.count`: the number of tokens after the token migration tool displays progress in the logs and, optionally, on the STDOUT. - modified the token generation page to accept the new DerbyDB TSS. ## How was this patch tested? Configured Knox to have the `AliasBasedTSS` as the token state backend and to allow unlimited token creation: ``` <property> <name>gateway.service.tokenstate.impl</name> <value>org.apache.knox.gateway.services.token.impl.AliasBasedTokenStateService</value> </property> <property> <name>gateway.knox.token.limit.per.user</name> <value>-1</value> </property> ``` Generated 456 tokens with random expiration times (456x4=1824 aliases) then stopped the Knox GW (to avoid the reaper thread removing expired tokens). ``` $ bin/knoxcli.sh list-alias | grep "items" 1827 ``` Executed the new KnoxCLI command to confirm it only migrates anything if the configured backend allows token migration: ``` $ bin/knoxcli.sh migrate-tokens --progressCount 15 --archiveMigrated true This tool is meant to migrate tokens into a JDBC TokenStateService backend. However, the currently configured one (org.apache.knox.gateway.services.token.impl.AliasBasedTokenStateService) does not fulfill this requirement! ``` Before running the new KnoxLCI command again, I commented out the `gateway.service.tokenstate.impl` param in `gateway-site.xml` => the new default, DerbyDBTSS, was in place. Executed the command again: ``` $ bin/knoxcli.sh migrate-tokens --progressCount 15 --archiveMigrated true > ~/migrationResultWithArchival.txt $ bin/knoxcli.sh list-alias | grep items 3 items. $ bin/knoxcli.sh list-alias --cluster __tokens | grep items 1824 items. $ cat ~/migrationResultWithArchival.txt Migrating tokens from __gateway credential store into the configured TokenStateService backend... Loading token aliases from the __gateway credential store. This could take a while. Token aliased loaded in 178741 milliseconds Processed 15 tokens in 102 milliseconds Processed 30 tokens in 174 milliseconds ... Processed 450 tokens in 2191 milliseconds Processed 456 tokens in 2202 milliseconds Archiving token aliases in the __tokens credential store... Archived token related aliases in the __tokens credential store in 141849 millsieconds Removing token aliases from the __gateway credential store... Removed token related aliases from the __gateway credential store in 38 milliseconds ``` --- Repeated the generate/migration step, but this time without token archival: ``` $ bin/knoxcli.sh migrate-tokens --progressCount 15 > ~/migrationResultWithoutArchival.txt $ cat ~/migrationResultWithoutArchival.txt Migrating tokens from __gateway credential store into the configured TokenStateService backend... Loading token aliases from the __gateway credential store. This could take a while. Token aliased loaded in 182497 milliseconds Processed 15 tokens in 160 milliseconds Processed 30 tokens in 271 milliseconds ... Processed 456 tokens in 1677 milliseconds Removing token aliases from the __gateway credential store... Removed token related aliases from the __gateway credential store in 61 milliseconds ``` I also tested the token migration tool integration during the Knox Gateway startup. I removed the previously created data/security/tokens folder, switched to AliasBasedTSS and created another 456 tokens. Then switched back to the default DerbyDBTSS and started the Knox GW: ``` 2023-12-14 10:14:18,653 INFO knox.gateway (GatewayServer.java:logSysProp(227)) - System Property: user.name=sandormolnar 2023-12-14 10:14:18,659 INFO knox.gateway (GatewayServer.java:logSysProp(227)) - System Property: user.dir=/Users/sandormolnar/test/knoxGateway 2023-12-14 10:14:18,659 INFO knox.gateway (GatewayServer.java:logSysProp(227)) - System Property: java.runtime.name=OpenJDK Runtime Environment 2023-12-14 10:14:18,659 INFO knox.gateway (GatewayServer.java:logSysProp(227)) - System Property: java.runtime.version=1.8.0_282-bre_2021_01_20_16_37-b00 2023-12-14 10:14:18,659 INFO knox.gateway (GatewayServer.java:logSysProp(227)) - System Property: java.home=/usr/local/Cellar/openjdk@8/1.8.0+282/libexec/openjdk.jdk/Contents/Home/jre ... 2023-12-14 10:14:22,001 INFO knox.gateway (AbstractServiceFactory.java:logServiceUsage(103)) - Using org.apache.knox.gateway.services.token.impl.DerbyDBTokenStateService implementation for TokenStateService ... 2023-12-14 10:14:24,932 INFO knox.gateway (AbstractGatewayServices.java:start(60)) - Starting service: org.apache.knox.gateway.services.token.impl.DerbyDBTokenStateService 2023-12-14 10:14:24,937 INFO token.state (TokenMigrationTool.java:log(114)) - Loading token aliases from the __gateway credential store. This could take a while. 2023-12-14 10:17:35,276 INFO token.state (TokenMigrationTool.java:log(114)) - Token aliases loaded in 190342 milliseconds 2023-12-14 10:17:35,514 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token cdb286ff-2037-41ea-8918-c338675f92eb into the configured TokenStateService backend. 2023-12-14 10:17:35,515 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = bd11f6bf-6ee4-4322-98ab-6797f8ac7d00 2023-12-14 10:17:35,562 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 5b8a754f-78c7-428a-95d6-ff6cff586b87 into the configured TokenStateService backend. 2023-12-14 10:17:35,601 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token d0281d9c-a652-4d79-824e-7b53f4476ea2 into the configured TokenStateService backend. 2023-12-14 10:17:35,634 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token a4d506d8-28aa-41be-8a45-ee9e3f7d109c into the configured TokenStateService backend. 2023-12-14 10:17:35,634 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 7d7a24b1-fcba-4828-9bc4-a22623a78357 2023-12-14 10:17:35,660 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 0c52b28b-b6d3-411a-a8af-d148a05aac0a into the configured TokenStateService backend. 2023-12-14 10:17:35,684 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token d94263f1-5e5b-4fe3-9d78-74a26caee446 into the configured TokenStateService backend. 2023-12-14 10:17:35,684 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 72e7bb73-d53b-4e89-9a8b-d5c4a17a93b3 2023-12-14 10:17:35,684 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 7f89e3c5-4567-40c7-b713-17e54695cb6c 2023-12-14 10:17:35,685 INFO token.state (TokenMigrationTool.java:log(114)) - Processed 10 tokens in 342 milliseconds 2023-12-14 10:17:35,708 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 3dbe9224-ac82-43f1-b383-efd930dcfb09 into the configured TokenStateService backend. 2023-12-14 10:17:35,731 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 760c1ff9-49f6-4ba0-9114-2c0dd3cdc9d2 into the configured TokenStateService backend. 2023-12-14 10:17:35,753 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token d67cc4ab-4de0-4f2d-8664-3e9dd721b634 into the configured TokenStateService backend. 2023-12-14 10:17:35,775 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 49a55ad7-44f6-48cc-bd4a-3db5a2108029 into the configured TokenStateService backend. 2023-12-14 10:17:35,775 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 853527ab-6c52-4d8c-8dd6-f7cf5c4f3b27 2023-12-14 10:17:35,776 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 681206a9-3335-4129-90c3-50f996f194c1 2023-12-14 10:17:35,798 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token f424005f-2ae0-498c-949b-760c880e4662 into the configured TokenStateService backend. 2023-12-14 10:17:35,799 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 8ab5fc0f-02d6-44d6-89ec-d41b4825c29d 2023-12-14 10:17:35,829 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 772e0455-4913-48ff-b66d-422e7d624f43 into the configured TokenStateService backend. 2023-12-14 10:17:35,856 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 8b24070d-11f6-47e9-8832-faaee12faed0 into the configured TokenStateService backend. 2023-12-14 10:17:35,857 INFO token.state (TokenMigrationTool.java:log(114)) - Processed 20 tokens in 515 milliseconds 2023-12-14 10:17:35,857 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 1fc77b80-c96d-4f3e-9841-cd2a2703e2db 2023-12-14 10:17:35,857 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 4dd4955e-1380-4f12-9b5d-49052e4c794b 2023-12-14 10:17:35,882 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token dfa2ca51-2f9c-4ebf-95c1-760c1ed23d89 into the configured TokenStateService backend. 2023-12-14 10:17:35,908 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 48a363c5-7abc-4914-ab11-84bd8caf7423 into the configured TokenStateService backend. 2023-12-14 10:17:35,909 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 3a5888a1-d44a-4f72-b100-bbeeccc11f2b 2023-12-14 10:17:35,909 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = ca972a16-0458-4a07-b547-26d2e8361a0e 2023-12-14 10:17:35,909 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 6d132cf8-9307-4b53-9931-7ba950529850 2023-12-14 10:17:35,909 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = cdbba2df-8884-4caf-a693-a1ea5268f48f 2023-12-14 10:17:35,934 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 326e575a-6fea-4cc8-957c-e5829d76f517 into the configured TokenStateService backend. 2023-12-14 10:17:35,954 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 31f7daee-e134-49e8-a633-4ec9e38bce57 into the configured TokenStateService backend. 2023-12-14 10:17:35,954 INFO token.state (TokenMigrationTool.java:log(114)) - Processed 30 tokens in 612 milliseconds ... 2023-12-14 10:17:40,306 INFO token.state (TokenMigrationTool.java:log(114)) - Processed 440 tokens in 4964 milliseconds 2023-12-14 10:17:40,306 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 50439317-05a7-42ef-8dbc-0f4a648d4c46 2023-12-14 10:17:40,306 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 95332d90-60db-4d54-9b3f-202d2de3f7b4 2023-12-14 10:17:40,306 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = bca3afa2-f1f0-44ae-8766-c938a624e949 2023-12-14 10:17:40,317 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 7d3347c9-c059-42f6-b663-ab8964654010 into the configured TokenStateService backend. 2023-12-14 10:17:40,328 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 6a879abc-03c8-42d6-b990-d2d61d45291c into the configured TokenStateService backend. 2023-12-14 10:17:40,339 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token c7a298bd-b3df-418c-b235-3ae90dcfd1b4 into the configured TokenStateService backend. 2023-12-14 10:17:40,358 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 05a4d692-0618-4008-a7b2-981bcb75a5fe into the configured TokenStateService backend. 2023-12-14 10:17:40,374 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 84143f53-11a2-4d94-8f86-84971a2d1905 into the configured TokenStateService backend. 2023-12-14 10:17:40,388 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 69787bc1-2361-4e90-a219-bf132bea6cce into the configured TokenStateService backend. 2023-12-14 10:17:40,403 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 7c855052-9a69-4f52-8ee0-f8b952ecb656 into the configured TokenStateService backend. 2023-12-14 10:17:40,403 INFO token.state (TokenMigrationTool.java:log(114)) - Processed 450 tokens in 5061 milliseconds 2023-12-14 10:17:40,418 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 9b1ee5aa-9600-4fac-9f48-f152c01dd94b into the configured TokenStateService backend. 2023-12-14 10:17:40,438 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token c03c4748-98c6-4e83-a399-f520e8d579b3 into the configured TokenStateService backend. 2023-12-14 10:17:40,455 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 5bbb60ba-7608-4437-b4cc-72c81097bf8a into the configured TokenStateService backend. 2023-12-14 10:17:40,455 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = 6da78f70-22d2-4f52-b9e7-f274d2028468 2023-12-14 10:17:40,455 INFO token.state (TokenMigrationTool.java:log(114)) - Skipping the migration of expired token with ID = e7b5e8a2-829a-40a2-8cd4-150943d69788 2023-12-14 10:17:40,474 INFO token.state (TokenMigrationTool.java:log(114)) - Migrated token 1419101b-3714-4e32-9a3b-98ffa46a0e3e into the configured TokenStateService backend. 2023-12-14 10:17:40,474 INFO token.state (TokenMigrationTool.java:log(114)) - Processed 456 tokens in 5132 milliseconds 2023-12-14 10:17:40,474 INFO token.state (TokenMigrationTool.java:log(114)) - Removing token aliases from the __gateway credential store... 2023-12-14 10:17:40,536 INFO token.state (TokenMigrationTool.java:log(114)) - Removed token related aliases from the __gateway credential store in 62 milliseconds ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@knox.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
