> On Dec. 21, 2021, 7:09 p.m., Yao Zhou wrote: > > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java > > Lines 136-166 (patched) > > <https://reviews.apache.org/r/73756/diff/4/?file=2257093#file2257093line136> > > > > Fetching all streams from a given log group could be quite slow and can > > easily trigger throttling given every AWS account has API limits. > > > > 1/ Have you verified that filterLogEvents() does not work if we don't > > provide a log stream? I was able to do it through AWS CLI but haven't tried > > SDK yet. > > ``` > > aws logs filter-log-events --log-group-name "xxx" > > > > ....(all events across streams) > > ``` > > 2/ I would suggest we change the log stream to log stream prefix, which > > matches the client-side config and also gives you the benefit of searching > > events across streams (instead of withLogStreamNames, you do > > withLogStreamPrefix see > > https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/logs/model/FilterLogEventsRequest.html#setLogStreamNamePrefix-java.lang.String-) > > > > ``` > > logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + > > "log_stream"); > > ``` > > Pradeep Agrawal wrote: > I tried with below given diff and its not returning any records. I added > debug logs also and its not returning anything. > > ======== > > diff --git > a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java > > b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java > index 896ed05e1..cc300eacb 100644 > --- > a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java > +++ > b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java > @@ -21,6 +21,7 @@ package org.apache.ranger.amazon.cloudwatch; > > import static > org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.CONFIG_PREFIX; > import static > org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_GROUP_NAME; > +import static > org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_STREAM_PREFIX; > > import java.text.SimpleDateFormat; > import java.util.ArrayList; > @@ -45,12 +46,9 @@ import > org.springframework.beans.factory.annotation.Autowired; > import org.springframework.stereotype.Component; > > import com.amazonaws.services.logs.AWSLogs; > -import com.amazonaws.services.logs.model.DescribeLogStreamsRequest; > -import com.amazonaws.services.logs.model.DescribeLogStreamsResult; > import com.amazonaws.services.logs.model.FilterLogEventsRequest; > import com.amazonaws.services.logs.model.FilterLogEventsResult; > import com.amazonaws.services.logs.model.FilteredLogEvent; > -import com.amazonaws.services.logs.model.LogStream; > > @Component > public class CloudWatchUtil { > @@ -66,7 +64,7 @@ public class CloudWatchUtil { > > public CloudWatchUtil() { > logGroupName = PropertiesUtil.getProperty(CONFIG_PREFIX + > "." + PROP_LOG_GROUP_NAME, "ranger_audits"); > - logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX > + "." + "log_stream"); > + logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX > + "." + PROP_LOG_STREAM_PREFIX, "ranger"); > String timeZone = > PropertiesUtil.getProperty("ranger.cloudwatch.timezone"); > if (timeZone != null) { > LOGGER.info("Setting timezone to " + timeZone); > @@ -131,7 +129,7 @@ public class CloudWatchUtil { > Date fromDate = null; > Date toDate = null; > > - String nextToken = null; > + /*String nextToken = null; > boolean done = false; > // load log stream names from cloudwatch if logStreamName > is not provided > List<String> logStreamNames = new ArrayList<String>(); > @@ -163,7 +161,7 @@ public class CloudWatchUtil { > } while (!done); > } else { > logStreamNames.add(logStreamName); > - } > + }*/ > > if (searchCriteria.getParamList() != null) { > List<String> filterExpr = new ArrayList<String>(); > @@ -238,7 +236,7 @@ public class CloudWatchUtil { > // Add FilterPattern which will only fetch logs required > filterLogEventsRequest = new FilterLogEventsRequest() > .withLogGroupName(logGroupName) > - .withLogStreamNames(logStreamNames) > + .withLogStreamNamePrefix(logStreamName) > .withStartTime(fromDate.getTime()) > .withEndTime(toDate.getTime()) > > .withFilterPattern(filterPattern.toString()); > > Pradeep Agrawal wrote: > Can you confirm the same at your end. > > Yao Zhou wrote: > I tested with below code and I was able to see events from different log > streams. The code is written in Scala but it's using the latest AWS JAVA SDK > (1.12.122). I suspect that you are seeing empty result because your filters > (e.g. startTime/endTime/filterPattern) are excluding all the events. > > ``` > val logs: AWSLogs = AWSLogsClientBuilder > .standard() > .withCredentials(credProvider) > .withRegion("us-east-1") > .build() > > val req = new FilterLogEventsRequest() > req.setLogGroupName("xxx") > val events: List[FilteredLogEvent] = > logs.filterLogEvents(req).getEvents.asScala.toList > events.foreach(println) > ```
Done. It seems there was jar conflict in my last attempt. Probably old jar was being referred somehow(though i deleted them) and was causing the issue. - Pradeep ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73756/#review223893 ----------------------------------------------------------- On Dec. 23, 2021, 6:56 a.m., Pradeep Agrawal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73756/ > ----------------------------------------------------------- > > (Updated Dec. 23, 2021, 6:56 a.m.) > > > Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay > Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, > and Velmurugan Periasamy. > > > Bugs: RANGER-3540 > https://issues.apache.org/jira/browse/RANGER-3540 > > > Repository: ranger > > > Description > ------- > > **Problem Statement: ** This is related to RANGER-2967 which includes changes > only at the plugin end. Access audit logs should be accessible and appear at > Ranger admin UI end as well. > > **Proposed Solution: ** Proposed patch make use of AWS API's to read access > audit logs from cloudwatch loggroup. > > **Known issue:** Cloudwatch APIs does not provide sorting of recording in > descending order of timestamp, hence read operation will be slow. Hence its > recommended to use the filter to minimise the resultset which shall reduce > the response time and access audit page will load faster. > Due to this issue as of now maximum 10k records will be loaded at a time to > handle out of memory issue. > > > Diffs > ----- > > agents-audit/pom.xml 5d031cca1 > > agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java > b236a2653 > agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java > f58b813f8 > hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41 > hbase-agent/scripts/install.properties 87a24819e > hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08 > hdfs-agent/scripts/install.properties 323b878cf > hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5 > hive-agent/scripts/install.properties 3720b66c8 > kms/scripts/install.properties 6b6b66270 > knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5 > knox-agent/scripts/install.properties 470400499 > plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f > plugin-atlas/scripts/install.properties 3b777bd6a > plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5 > plugin-elasticsearch/scripts/install.properties 4111afe3f > plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d > plugin-kafka/scripts/install.properties 1e325e0ec > plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489 > plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5 > plugin-kylin/scripts/install.properties 013433837 > plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29 > plugin-ozone/scripts/install.properties 1891d565f > plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d > plugin-presto/scripts/install.properties ce162a2bd > plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696 > plugin-solr/scripts/install.properties d1852e695 > plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5 > plugin-sqoop/scripts/install.properties 81b4526a6 > plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5 > plugin-yarn/scripts/install.properties e73ab8b14 > pom.xml f9c46f669 > security-admin/pom.xml e9e9a537b > security-admin/scripts/install.properties 5a8b00c13 > security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf > security-admin/scripts/setup.sh c3f51a03a > security-admin/scripts/upgrade_admin.py 10fa485bd > security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java > 4d97f28fd > > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158 > security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java > 75ebae6f5 > security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b > > security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java > 0b2e7df7f > > security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java > 9bee640a5 > > security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java > 0aea46d1b > security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f > security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec > storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5 > storm-agent/scripts/install.properties d219abf59 > > > Diff: https://reviews.apache.org/r/73756/diff/5/ > > > Testing > ------- > > Tested by creating IAM user in AWS and provided required configuration in the > install.properties. > > **Note:** AWS region name, access key and secret key should be provided in > the environment. > > > Thanks, > > Pradeep Agrawal > >
