>> Do you mean just for HDFS/File/Log4j or for DB and Solr also? It depends on how Solr expects the date field to be formatted in.
For HDFS/Log4j - we should publish the date format used to write audit log, so that the readers can be configured appropriately. Going forward, I would suggest using a format that includes the timezone. For backward compatibility sake, it should be possible to configure each destination to use the legacy format "yyyy-MM-dd HH:mm:ss.SSS” - a configuration like “xasecure.audit.destination.<type>.date.format.legacy=false". Thanks, Madhan On 2/1/16, 12:52 AM, "Gautam Borad" <[email protected]> wrote: >>>For JSON, how about including the timezone? > >Do you mean just for HDFS/File/Log4j or for DB and Solr also? Thanks. > > >On Mon, Feb 1, 2016 at 12:59 PM, Madhan Neethiraj < >[email protected]> wrote: > >> Gautam, >> >> >> Do we need to add timezone info also with event time, if yes then what >> should be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS". >> For JSON, how about including the timezone? Please find how/if the new >> format (that includes timezone) can co-exist with the earlier format? If >> not, then we would have to support writing in older format – depending upon >> a configuration; by default the audit log should be written with the >> timezone. >> >> Thanks, >> Madhan >> >> >> From: Gautam Borad <[email protected]> on behalf of Gautam Borad >> <[email protected]> >> Reply-To: Gautam Borad <[email protected]> >> Date: Thursday, January 28, 2016 at 5:13 AM >> To: Abhay Kulkarni <[email protected]>, Ramesh Mani < >> [email protected]>, Velmurugan Periasamy <[email protected]>, >> Alok Lal <[email protected]>, Selvamohan Neethiraj < >> [email protected]>, Madhan Neethiraj <[email protected]>, Don >> Bosco Durai <[email protected]> >> Cc: ranger <[email protected]>, Gautam Borad < >> [email protected]> >> Subject: Re: Review Request 42601: RANGER-798 - Approach 1 : Handle >> different timezone issue while saving audit logs to Solr >> >> This is an automatically generated e-mail. To reply, visit: >> https://reviews.apache.org/r/42601/ >> >> On January 28th, 2016, 8:32 a.m. UTC, *Madhan Neethiraj* wrote: >> >> >> agents-audit/src/main/java/org/apache/ranger/audit/destination/DBAuditDestination.java >> <https://reviews.apache.org/r/42601/diff/1/?file=1205404#file1205404line97>(Diff >> revision 1) >> >> public boolean log(Collection<AuditEventBase> events) { >> >> 97 >> >> >> eventCopy=cloneAuthzAuditEvent((AuthzAuditEvent)event); >> >> Instead of cloning and updating the eventDate in every destination >> (db/hdfs/solr/log4j), consider sending the "local" time (as set in >> AuthzAuditEvent.eventTime) to all audit destinations - exception DB, which >> require the time in UTC. For DB, consider updating the time in >> AuthzAuditEventDbObj(AuthzAuditEvent event) constructor. >> >> Do we need to add timezone info also with event time, if yes then what >> should be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS". >> >> >> - Gautam >> >> On January 28th, 2016, 1:12 p.m. UTC, Gautam Borad wrote: >> Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni, >> Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan >> Periasamy. >> By Gautam Borad. >> >> *Updated Jan. 28, 2016, 1:12 p.m.* >> *Bugs: *RANGER-798 <https://issues.apache.org/jira/browse/RANGER-798> >> *Repository: *ranger >> Description >> >> *Problem Statement*: >> The current implementation of “auditEvent.getEventTime()” contains time in >> UTC and since it's a date object it will contain component machine's local >> timezone. When Solr receives this date object and timezone, it tries to >> convert it from given timezone to UTC timestamp, which leads to double >> conversion of actual time before it get stored in Solr. >> >> *Proposed Solution*: >> If we can provide server local time and timezone to Solr then Solr will >> convert the received time from given timezone to UTC. >> As an alternate solution, replaced getUTCDate() with new Date() object at >> various places for audit event time, all audit destination will receive >> local Date object, for Solr there will be no conversion on received Date >> object but for all other audit destination we need to convert the received >> Date value to UTC timestamp as audit logs are being stored in UTC timestamp >> for all service/component. If all destination thread are enabled then >> changing the received event object may create issue in other audit >> destination as same event object is refferred everywhere. Hence received >> event object attributes value are being copied in another local event object >> and the updated event time can be stored there, after this local event >> object will be used to convert that in JSON to write in HDFS, or can be >> persisted in DB. >> >> Testing >> >> Steps performed(after patch) : >> 1) Changed plugin system time zone to IST and restarted all components. >> 2) Initiated an HDFS audit event. >> 3) Checked event time of newly created audit log in Solr, Audit log event >> time was matching with UTC. >> 4) Checked event time in Ranger UI, newly generated Audit event is matching >> with current time. >> 5) Checked event time of newly created audit log in xa_access_audit table, >> Audit log event time was matching with UTC. >> 6) Checked event time of newly created audit log in HDFS logs, Audit log >> event time was matching with UTC. >> >> Note: Will test other services audit logs after this approach is reviewed. >> >> Diffs >> >> - >> agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java >> (d52a60a) >> - >> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java >> (9586f73) >> - >> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java >> (fe50ca6) >> - >> hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java >> (5125af7) >> - >> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java >> (2ae4149) >> - >> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java >> (0f13577) >> - >> plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java >> (bb6a337) >> - >> plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java >> (04b8b91) >> - >> plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java >> (a8ecf15) >> - >> plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java >> (ab9b7a9) >> >> View Diff <https://reviews.apache.org/r/42601/diff/> >> > > > >-- >Regards, >Gautam.
