[
https://issues.apache.org/jira/browse/RANGER-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17887050#comment-17887050
]
Madhan Neethiraj commented on RANGER-4809:
------------------------------------------
Following shows use of the tool in docker setup to migrate 300k transaction
logs (100k policies create/update/delete) from x_trx_log to x_trx_log_v2.
Migration completed in 22 seconds.
{noformat}
$ export RANGER_ADMIN_HOME=/opt/ranger/admin/
$ export RANGER_ADMIN_CONF=/opt/ranger/admin/conf/
$ export RANGER_ADMIN_LOG_DIR=/var/log/ranger/
$ /opt/ranger/admin/ews/ranger-admin-transaction-log-migrate.sh
RANGER_ADMIN_HOME : /opt/ranger/admin/
RANGER_ADMIN_CONF : /opt/ranger/admin/conf/
RANGER_ADMIN_LOG_DIR : /var/log/ranger/
Using heap size 1g
Migrating Transaction logs has started with PID - 6281
$ tail -fF /var/log/ranger/ranger_db_patch.log
...
2024-10-04 23:10:56,062 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil
(TrxLogV2MigrationUtil.java:105) ==> TrxLogV2MigrationUtil.execLoad()
2024-10-04 23:10:56,062 [Loader Monitor] INFO
o.a.r.p.c.TrxLogV2MigrationUtil$Stats (TrxLogV2MigrationUtil.java:391)
PROGRESS: 0 of 0 transactions processed. Last migrated transaction(id=null,
time=null). Counts(migrated: 0, failed: 0, already-migrated: 0)
2024-10-04 23:10:56,062 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil
(TrxLogV2MigrationUtil.java:122) ==> TrxLogV2MigrationUtil.migrateTrxLogs()
2024-10-04 23:10:57,211 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil
(TrxLogV2MigrationUtil.java:148) Found 300097 transactions to migrate
2024-10-04 23:10:57,212 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil
(TrxLogV2MigrationUtil.java:150) Starting 5 threads to migrate, commit batch
size: 25
2024-10-04 23:10:57,440 [Thread-5] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 1000 of 300097 transactions
processed. Last migrated transaction(id=9403, time=2024/10/04 22:41:43 +0000).
Counts(migrated: 1000, failed: 0, already-migrated: 0)
2024-10-04 23:10:57,575 [Thread-4] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 2000 of 300097 transactions
processed. Last migrated transaction(id=19420, time=2024/10/04 22:41:51 +0000).
Counts(migrated: 2000, failed: 0, already-migrated: 0)
2024-10-04 23:10:57,712 [Thread-8] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 3000 of 300097 transactions
processed. Last migrated transaction(id=29690, time=2024/10/04 22:41:57 +0000).
Counts(migrated: 3000, failed: 0, already-migrated: 0)
2024-10-04 23:10:57,833 [Thread-7] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 4000 of 300097 transactions
processed. Last migrated transaction(id=39370, time=2024/10/04 22:42:06 +0000).
Counts(migrated: 4002, failed: 0, already-migrated: 0)
2024-10-04 23:10:57,948 [Thread-7] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 5000 of 300097 transactions
processed. Last migrated transaction(id=49337, time=2024/10/04 22:42:13 +0000).
Counts(migrated: 5000, failed: 0, already-migrated: 0)
2024-10-04 23:10:58,073 [Thread-7] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 6000 of 300097 transactions
processed. Last migrated transaction(id=59323, time=2024/10/04 22:42:19 +0000).
Counts(migrated: 6002, failed: 0, already-migrated: 0)
2024-10-04 23:10:58,189 [Thread-5] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 7000 of 300097 transactions
processed. Last migrated transaction(id=69550, time=2024/10/04 22:42:26 +0000).
Counts(migrated: 7000, failed: 0, already-migrated: 0)
2024-10-04 23:10:58,300 [Thread-8] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 8000 of 300097 transactions
processed. Last migrated transaction(id=79099, time=2024/10/04 22:42:33 +0000).
Counts(migrated: 8000, failed: 0, already-migrated: 0)
2024-10-04 23:10:58,427 [Thread-8] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 9000 of 300097 transactions
processed. Last migrated transaction(id=89303, time=2024/10/04 22:42:40 +0000).
Counts(migrated: 9000, failed: 0, already-migrated: 0)
2024-10-04 23:10:58,543 [Thread-6] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 10000 of 300097 transactions
processed. Last migrated transaction(id=98883, time=2024/10/04 22:42:46 +0000).
Counts(migrated: 10002, failed: 0, already-migrated: 0)
...
2024-10-04 23:11:17,929 [Thread-5] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 298000 of 300097 transactions
processed. Last migrated transaction(id=3063228, time=2024/10/04 23:04:42
+0000). Counts(migrated: 298001, failed: 0, already-migrated: 0)
2024-10-04 23:11:17,997 [Thread-6] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 299000 of 300097 transactions
processed. Last migrated transaction(id=3080792, time=2024/10/04 23:04:44
+0000). Counts(migrated: 299000, failed: 0, already-migrated: 0)
2024-10-04 23:11:18,067 [Thread-4] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 300000 of 300097 transactions
processed. Last migrated transaction(id=3099515, time=2024/10/04 23:04:46
+0000). Counts(migrated: 300000, failed: 0, already-migrated: 0)
2024-10-04 23:11:18,076 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil
(TrxLogV2MigrationUtil.java:164) <== TrxLogV2MigrationUtil.migrateTrxLogs()
2024-10-04 23:11:18,076 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil
(TrxLogV2MigrationUtil.java:113) <== TrxLogV2MigrationUtil.execLoad():
migration completed. Transaction counts(total: 300097, migrated: 300097,
already-migrated: 0, failed: 0)
2024-10-04 23:11:18,076 [main] INFO o.a.r.p.BaseLoader (BaseLoader.java:108)
###############################################
2024-10-04 23:11:18,076 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 300097 of 300097 transactions
processed. Last migrated transaction(id=3100371, time=2024/10/04 23:04:46
+0000). Counts(migrated: 300097, failed: 0, already-migrated: 0)
2024-10-04 23:11:18,076 [main] INFO o.a.r.p.BaseLoader (BaseLoader.java:110)
Loading completed!!!. Time taken=22secs, 14ms for 0
2024-10-04 23:11:18,076 [main] INFO o.a.r.p.BaseLoader (BaseLoader.java:113)
###############################################
2024-10-04 23:11:18,076 [Loader Monitor] INFO o.a.r.p.BaseLoader$1
(BaseLoader.java:140) Monitor Thread exiting!!!
2024-10-04 23:11:18,080 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil
(TrxLogV2MigrationUtil.java:82) Load complete. Exiting!!!
{noformat}
> Utility to migrate admin audit logs in x_trx_log table x_trx_log_v2 table
> -------------------------------------------------------------------------
>
> Key: RANGER-4809
> URL: https://issues.apache.org/jira/browse/RANGER-4809
> Project: Ranger
> Issue Type: Improvement
> Components: admin
> Reporter: Madhan Neethiraj
> Assignee: Rakesh Gupta
> Priority: Major
> Fix For: 3.0.0, 2.6.0
>
>
> With updates in RANGER-4803, Ranger will store admin audit records in
> {{x_trx_log_v2}} table. Prior to this, {{x_trx_log}} table was used to store
> these records. To make earlier admin audits available to updated Ranger,
> existing entries in {{x_trx_log}} should be migrated to {{x_trx_log_v2}}
> table. Here are few notes to take into account:
> # For a given admin action, like create/update/delete of a policy, multiple
> rows are created in {{x_trx_log}} table - one row for each updated attribute.
> # For a given admin action, a single row is created in {{x_trx_log_v2}}
> table. This row captures details of all updated attributes in json format.
> # Migration utility should merge all rows for a given admin action,
> identified by column {{x_trx_log.trx_id}}, into a single record in
> {{x_trx_log_v2}} table. {{x_trx_log_v2.change_info}} column should be
> populated with contents of following columns in multiple {{x_trx_log}} rows:
> {{attr_name}}, {{prev_val}}, {{new_val}}.
> # The utility should provide an option to migrate only a subset of entries in
> {{x_trx_log}} table - for example, for a given time period (last 3 months,
> last 1 year). This will help avoid migrating older records that are of no
> interest to the customer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
