>>For JSON, how about including the timezone? Do you mean just for HDFS/File/Log4j or for DB and Solr also? Thanks.
On Mon, Feb 1, 2016 at 12:59 PM, Madhan Neethiraj < [email protected]> wrote: > Gautam, > > >> Do we need to add timezone info also with event time, if yes then what > should be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS". > For JSON, how about including the timezone? Please find how/if the new > format (that includes timezone) can co-exist with the earlier format? If > not, then we would have to support writing in older format – depending upon > a configuration; by default the audit log should be written with the > timezone. > > Thanks, > Madhan > > > From: Gautam Borad <[email protected]> on behalf of Gautam Borad > <[email protected]> > Reply-To: Gautam Borad <[email protected]> > Date: Thursday, January 28, 2016 at 5:13 AM > To: Abhay Kulkarni <[email protected]>, Ramesh Mani < > [email protected]>, Velmurugan Periasamy <[email protected]>, > Alok Lal <[email protected]>, Selvamohan Neethiraj < > [email protected]>, Madhan Neethiraj <[email protected]>, Don > Bosco Durai <[email protected]> > Cc: ranger <[email protected]>, Gautam Borad < > [email protected]> > Subject: Re: Review Request 42601: RANGER-798 - Approach 1 : Handle > different timezone issue while saving audit logs to Solr > > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/42601/ > > On January 28th, 2016, 8:32 a.m. UTC, *Madhan Neethiraj* wrote: > > > agents-audit/src/main/java/org/apache/ranger/audit/destination/DBAuditDestination.java > <https://reviews.apache.org/r/42601/diff/1/?file=1205404#file1205404line97>(Diff > revision 1) > > public boolean log(Collection<AuditEventBase> events) { > > 97 > > > eventCopy=cloneAuthzAuditEvent((AuthzAuditEvent)event); > > Instead of cloning and updating the eventDate in every destination > (db/hdfs/solr/log4j), consider sending the "local" time (as set in > AuthzAuditEvent.eventTime) to all audit destinations - exception DB, which > require the time in UTC. For DB, consider updating the time in > AuthzAuditEventDbObj(AuthzAuditEvent event) constructor. > > Do we need to add timezone info also with event time, if yes then what should > be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS". > > > - Gautam > > On January 28th, 2016, 1:12 p.m. UTC, Gautam Borad wrote: > Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni, > Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan > Periasamy. > By Gautam Borad. > > *Updated Jan. 28, 2016, 1:12 p.m.* > *Bugs: *RANGER-798 <https://issues.apache.org/jira/browse/RANGER-798> > *Repository: *ranger > Description > > *Problem Statement*: > The current implementation of “auditEvent.getEventTime()” contains time in > UTC and since it's a date object it will contain component machine's local > timezone. When Solr receives this date object and timezone, it tries to > convert it from given timezone to UTC timestamp, which leads to double > conversion of actual time before it get stored in Solr. > > *Proposed Solution*: > If we can provide server local time and timezone to Solr then Solr will > convert the received time from given timezone to UTC. > As an alternate solution, replaced getUTCDate() with new Date() object at > various places for audit event time, all audit destination will receive > local Date object, for Solr there will be no conversion on received Date > object but for all other audit destination we need to convert the received > Date value to UTC timestamp as audit logs are being stored in UTC timestamp > for all service/component. If all destination thread are enabled then > changing the received event object may create issue in other audit > destination as same event object is refferred everywhere. Hence received > event object attributes value are being copied in another local event object > and the updated event time can be stored there, after this local event object > will be used to convert that in JSON to write in HDFS, or can be persisted in > DB. > > Testing > > Steps performed(after patch) : > 1) Changed plugin system time zone to IST and restarted all components. > 2) Initiated an HDFS audit event. > 3) Checked event time of newly created audit log in Solr, Audit log event > time was matching with UTC. > 4) Checked event time in Ranger UI, newly generated Audit event is matching > with current time. > 5) Checked event time of newly created audit log in xa_access_audit table, > Audit log event time was matching with UTC. > 6) Checked event time of newly created audit log in HDFS logs, Audit log > event time was matching with UTC. > > Note: Will test other services audit logs after this approach is reviewed. > > Diffs > > - > agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java > (d52a60a) > - agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java > (9586f73) > - > agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java > (fe50ca6) > - > hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java > (5125af7) > - > hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java > (2ae4149) > - > hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java > (0f13577) > - > plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java > (bb6a337) > - > plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java > (04b8b91) > - > plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java > (a8ecf15) > - > plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java > (ab9b7a9) > > View Diff <https://reviews.apache.org/r/42601/diff/> > -- Regards, Gautam.
