> On Dec. 21, 2021, 7:09 p.m., Yao Zhou wrote:
> > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> > Lines 136-166 (patched)
> > <https://reviews.apache.org/r/73756/diff/4/?file=2257093#file2257093line136>
> >
> > Fetching all streams from a given log group could be quite slow and can
> > easily trigger throttling given every AWS account has API limits.
> >
> > 1/ Have you verified that filterLogEvents() does not work if we don't
> > provide a log stream? I was able to do it through AWS CLI but haven't tried
> > SDK yet.
> > ```
> > aws logs filter-log-events --log-group-name "xxx"
> >
> > ....(all events across streams)
> > ```
> > 2/ I would suggest we change the log stream to log stream prefix, which
> > matches the client-side config and also gives you the benefit of searching
> > events across streams (instead of withLogStreamNames, you do
> > withLogStreamPrefix see
> > https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/logs/model/FilterLogEventsRequest.html#setLogStreamNamePrefix-java.lang.String-)
> >
> > ```
> > logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." +
> > "log_stream");
> > ```
I tried with below given diff and its not returning any records. I added debug
logs also and its not returning anything.
========
diff --git
a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
index 896ed05e1..cc300eacb 100644
---
a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
+++
b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
@@ -21,6 +21,7 @@ package org.apache.ranger.amazon.cloudwatch;
import static
org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.CONFIG_PREFIX;
import static
org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_GROUP_NAME;
+import static
org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_STREAM_PREFIX;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
@@ -45,12 +46,9 @@ import
org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import com.amazonaws.services.logs.AWSLogs;
-import com.amazonaws.services.logs.model.DescribeLogStreamsRequest;
-import com.amazonaws.services.logs.model.DescribeLogStreamsResult;
import com.amazonaws.services.logs.model.FilterLogEventsRequest;
import com.amazonaws.services.logs.model.FilterLogEventsResult;
import com.amazonaws.services.logs.model.FilteredLogEvent;
-import com.amazonaws.services.logs.model.LogStream;
@Component
public class CloudWatchUtil {
@@ -66,7 +64,7 @@ public class CloudWatchUtil {
public CloudWatchUtil() {
logGroupName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." +
PROP_LOG_GROUP_NAME, "ranger_audits");
- logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "."
+ "log_stream");
+ logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "."
+ PROP_LOG_STREAM_PREFIX, "ranger");
String timeZone =
PropertiesUtil.getProperty("ranger.cloudwatch.timezone");
if (timeZone != null) {
LOGGER.info("Setting timezone to " + timeZone);
@@ -131,7 +129,7 @@ public class CloudWatchUtil {
Date fromDate = null;
Date toDate = null;
- String nextToken = null;
+ /*String nextToken = null;
boolean done = false;
// load log stream names from cloudwatch if logStreamName is
not provided
List<String> logStreamNames = new ArrayList<String>();
@@ -163,7 +161,7 @@ public class CloudWatchUtil {
} while (!done);
} else {
logStreamNames.add(logStreamName);
- }
+ }*/
if (searchCriteria.getParamList() != null) {
List<String> filterExpr = new ArrayList<String>();
@@ -238,7 +236,7 @@ public class CloudWatchUtil {
// Add FilterPattern which will only fetch logs required
filterLogEventsRequest = new FilterLogEventsRequest()
.withLogGroupName(logGroupName)
- .withLogStreamNames(logStreamNames)
+ .withLogStreamNamePrefix(logStreamName)
.withStartTime(fromDate.getTime())
.withEndTime(toDate.getTime())
.withFilterPattern(filterPattern.toString());
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223893
-----------------------------------------------------------
On Dec. 10, 2021, 8:08 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2021, 8:08 p.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay
> Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu,
> and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes
> only at the plugin end. Access audit logs should be accessible and appear at
> Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access
> audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in
> descending order of timestamp, hence read operation will be slow. Hence its
> recommended to use the filter to minimise the resultset which shall reduce
> the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to
> handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
>
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java
> b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java
> f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java
> e902e65d0
>
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java
> PRE-CREATION
>
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java
> PRE-CREATION
>
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java
> 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
>
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java
> 0b2e7df7f
>
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java
> 9bee640a5
>
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java
> 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/4/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the
> install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in
> the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
