> On Dec. 15, 2021, 8:46 p.m., Yao Zhou wrote: > > agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java > > Line 159 (original), 176 (patched) > > <https://reviews.apache.org/r/73756/diff/1/?file=2256539#file2256539line177> > > > > DescribeLogStreams API supports pagination hence you may not get the > > stream by just calling it once. > > > > I saw you changed the log stream name to get rid of the UUID. There > > were two reasons behind it. > > > > 1/ We have a use case that one engine/service would generate audit > > events not only on master nodes but also on core nodes hence we prefer an > > unique stream for each node. > > > > 2/ By appending the UUID, we no longer need to pre-check if the log > > stream exists when ranger plugin gets restarted on the same machine. > > Pradeep Agrawal wrote: > I am reverting my changes here. > > > Also is it true that at every restart a new stream will be created ?
Yes. The init() method gets invoked whenever the plugin gets re-loaded (i.e. service restart) hence always creates a new log stream. > On Dec. 15, 2021, 8:46 p.m., Yao Zhou wrote: > > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java > > Lines 137-141 (patched) > > <https://reviews.apache.org/r/73756/diff/1/?file=2256578#file2256578line137> > > > > same comment: The DescribeLogStreams API supports pagination which > > needs to be handled. > > Pradeep Agrawal wrote: > Thanks for review. > How can we display audit logs in the ranger ui by consolidating them from > so many streams, also it seems the cloudwatch logs APIs does not provide > sorting of logs in descending order of timestamp. Q: How can we display audit logs in the ranger ui by consolidating them from so many streams? A: The API you are using should support searching across all the streams under a log group by not putting the log stream input (only log group). Q: it seems the cloudwatch logs APIs does not provide sorting of logs in descending order of timestamp A: The returned log events are sorted by event timestamp (needs to verify descending vs ascending). Ref: https://docs.aws.amazon.com/cli/latest/reference/logs/filter-log-events.html - Yao ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73756/#review223859 ----------------------------------------------------------- On Dec. 10, 2021, 8:08 p.m., Pradeep Agrawal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73756/ > ----------------------------------------------------------- > > (Updated Dec. 10, 2021, 8:08 p.m.) > > > Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay > Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, > and Velmurugan Periasamy. > > > Bugs: RANGER-3540 > https://issues.apache.org/jira/browse/RANGER-3540 > > > Repository: ranger > > > Description > ------- > > **Problem Statement: ** This is related to RANGER-2967 which includes changes > only at the plugin end. Access audit logs should be accessible and appear at > Ranger admin UI end as well. > > **Proposed Solution: ** Proposed patch make use of AWS API's to read access > audit logs from cloudwatch loggroup. > > **Known issue:** Cloudwatch APIs does not provide sorting of recording in > descending order of timestamp, hence read operation will be slow. Hence its > recommended to use the filter to minimise the resultset which shall reduce > the response time and access audit page will load faster. > Due to this issue as of now maximum 10k records will be loaded at a time to > handle out of memory issue. > > > Diffs > ----- > > agents-audit/pom.xml 5d031cca1 > > agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java > b236a2653 > agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java > f58b813f8 > hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41 > hbase-agent/scripts/install.properties 87a24819e > hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08 > hdfs-agent/scripts/install.properties 323b878cf > hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5 > hive-agent/scripts/install.properties 3720b66c8 > kms/scripts/install.properties 6b6b66270 > knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5 > knox-agent/scripts/install.properties 470400499 > plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f > plugin-atlas/scripts/install.properties 3b777bd6a > plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5 > plugin-elasticsearch/scripts/install.properties 4111afe3f > plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d > plugin-kafka/scripts/install.properties 1e325e0ec > plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489 > plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5 > plugin-kylin/scripts/install.properties 013433837 > plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29 > plugin-ozone/scripts/install.properties 1891d565f > plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d > plugin-presto/scripts/install.properties ce162a2bd > plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696 > plugin-solr/scripts/install.properties d1852e695 > plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5 > plugin-sqoop/scripts/install.properties 81b4526a6 > plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5 > plugin-yarn/scripts/install.properties e73ab8b14 > pom.xml f9c46f669 > security-admin/pom.xml e9e9a537b > security-admin/scripts/install.properties 5a8b00c13 > security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf > security-admin/scripts/setup.sh c3f51a03a > security-admin/scripts/upgrade_admin.py 10fa485bd > security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java > e902e65d0 > > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158 > security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java > 75ebae6f5 > security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b > > security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java > 0b2e7df7f > > security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java > 9bee640a5 > > security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java > 0aea46d1b > security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f > security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec > storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5 > storm-agent/scripts/install.properties d219abf59 > > > Diff: https://reviews.apache.org/r/73756/diff/4/ > > > Testing > ------- > > Tested by creating IAM user in AWS and provided required configuration in the > install.properties. > > **Note:** AWS region name, access key and secret key should be provided in > the environment. > > > Thanks, > > Pradeep Agrawal > >
