Re: Review Request 59828: Ability to anonymize data during log processing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59828/ --- (Updated June 19, 2017, 10:34 a.m.) Review request for Ambari, Oliver Szabo and Robert Nettleton. Changes --- fix description Bugs: AMBARI-21181 https://issues.apache.org/jira/browse/AMBARI-21181 Repository: ambari Description (updated) --- Have a new mapper called map_anonymize which accepts a pattern containing elements. After the mapping if the pattern is found in the field, than the parts matching the elements are replaced with a configurable hiding character ( default '*' ). Example: pattern: the secret is / which is confidential log message: Bla bla bla the secret is SECRET1 / SECRET2 which is confidential bla bla after the mapping: Bla bla bla the secret is *** / *** which is confidential bla bla This patch also fixes the checkpoint file writing, which wasn't atomic. The checkpoint files are now written under a temporary name first, than atomically renamed to the proper name, thus making it impossible to have a half written file. Diffs - ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapAnonymizeDescriptor.java PRE-CREATION ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapDateDescriptor.java f88435f ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapFieldDescriptor.java db086c5 ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapFieldValueDescriptor.java cf37e62 ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/inputconfig/impl/MapAnonymizeDescriptorImpl.java PRE-CREATION ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/inputconfig/impl/PostMapValuesAdapter.java 32aded8 ambari-logsearch/ambari-logsearch-logfeeder/docs/postMapValues.md 906359a ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java 2bc4e68 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java fba596d ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java 8c76785 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputS3File.java 4bf162b ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/mapper/MapperAnonymize.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java ffd6cec ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/alias_config.json e2ed625 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/mapper/MapperAnonymizeTest.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/test/resources/log4j.xml e641018 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapDate.java dcb0393 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapField.java b18439c ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapFieldAnonymize.java PRE-CREATION ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerPostMapValuesListDeserializer.java 18744e2 ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-hst.json 4ccef74 ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-zookeeper.json d3685a4 Diff: https://reviews.apache.org/r/59828/diff/2/ Testing --- Tested on local cluster, and by the new unit tests added. Thanks, Miklos Gergely
Re: Review Request 59828: Ability to anonymize data during log processing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59828/ --- (Updated June 19, 2017, 10:31 a.m.) Review request for Ambari, Oliver Szabo and Robert Nettleton. Changes --- Use * instead of X by default to hide sensitive log message parts Bugs: AMBARI-21181 https://issues.apache.org/jira/browse/AMBARI-21181 Repository: ambari Description (updated) --- Have a new mapper called map_anonymize which accepts a pattern containing elements. After the mapping if the pattern is found in the field, than the parts matching the elements are replaced with a configurable hiding character ( default '*' ). Example: pattern: the secret is / which is confidential log message: Bla bla bla the secret is SECRET1 / SECRET2 which is confidential bla bla after the mapping: Bla bla bla the secret is *** / *** which is confidential bla bla This patch also fixes the checkpoint file writing, which wasn't atomic. The checkpoint files are now written under a temporary name first, than atomically renamed to the proper name, thus making it impossible to have a half written file. Diffs (updated) - ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapAnonymizeDescriptor.java PRE-CREATION ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapDateDescriptor.java f88435f ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapFieldDescriptor.java db086c5 ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapFieldValueDescriptor.java cf37e62 ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/inputconfig/impl/MapAnonymizeDescriptorImpl.java PRE-CREATION ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/inputconfig/impl/PostMapValuesAdapter.java 32aded8 ambari-logsearch/ambari-logsearch-logfeeder/docs/postMapValues.md 906359a ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java 2bc4e68 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java fba596d ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java 8c76785 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputS3File.java 4bf162b ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/mapper/MapperAnonymize.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java ffd6cec ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/alias_config.json e2ed625 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/mapper/MapperAnonymizeTest.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/test/resources/log4j.xml e641018 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapDate.java dcb0393 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapField.java b18439c ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapFieldAnonymize.java PRE-CREATION ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerPostMapValuesListDeserializer.java 18744e2 ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-hst.json 4ccef74 ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-zookeeper.json d3685a4 Diff: https://reviews.apache.org/r/59828/diff/2/ Changes: https://reviews.apache.org/r/59828/diff/1-2/ Testing --- Tested on local cluster, and by the new unit tests added. Thanks, Miklos Gergely
Re: Review Request 59828: Ability to anonymize data during log processing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59828/#review177487 --- Ship it! The patch looks fine. One thing that may be worth considering for a future patch is the default hide character. It might be good to use * as the default, since password fields in UIs are typically replaced with * . That being said, I wouldn't necessarily change this patch to update the default hide character, but perhaps it should be changed in a different patch. - Robert Nettleton On June 6, 2017, 8:55 a.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59828/ > --- > > (Updated June 6, 2017, 8:55 a.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21181 > https://issues.apache.org/jira/browse/AMBARI-21181 > > > Repository: ambari > > > Description > --- > > Have a new mapper called map_anonymize which accepts a pattern containing > elements. After the mapping if the pattern is found in the field, than > the parts matching the elements are replaced with a configurable > hiding character ( default 'X' ). > > Example: > pattern: the secret is / which is confidential > log message: Bla bla bla the secret is SECRET1 / SECRET2 which is > confidential bla bla > after the mapping: Bla bla bla the secret is XXX / XXX which is > confidential bla bla > > This patch also fixes the checkpoint file writing, which wasn't atomic. The > checkpoint files are now written under a temporary name first, than > atomically renamed to the proper name, thus making it impossible to have a > half written file. > > > Diffs > - > > > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapAnonymizeDescriptor.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapDateDescriptor.java > f88435f > > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapFieldDescriptor.java > db086c5 > > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapFieldValueDescriptor.java > cf37e62 > > ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/inputconfig/impl/MapAnonymizeDescriptorImpl.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/inputconfig/impl/PostMapValuesAdapter.java > 32aded8 > ambari-logsearch/ambari-logsearch-logfeeder/docs/postMapValues.md 906359a > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java > 2bc4e68 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java > fba596d > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java > 8c76785 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputS3File.java > 4bf162b > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/mapper/MapperAnonymize.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java > ffd6cec > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/alias_config.json > e2ed625 > > ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/mapper/MapperAnonymizeTest.java > PRE-CREATION > ambari-logsearch/ambari-logsearch-logfeeder/src/test/resources/log4j.xml > e641018 > > ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapDate.java > dcb0393 > > ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapField.java > b18439c > > ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapFieldAnonymize.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerPostMapValuesListDeserializer.java > 18744e2 > > ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-hst.json > 4ccef74 > > ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-zookeeper.json > d3685a4 > > > Diff: https://reviews.apache.org/r/59828/di
Re: Review Request 59828: Ability to anonymize data during log processing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59828/#review177007 --- Ship it! Ship It! - Oliver Szabo On June 6, 2017, 8:55 a.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59828/ > --- > > (Updated June 6, 2017, 8:55 a.m.) > > > Review request for Ambari, Oliver Szabo and Robert Nettleton. > > > Bugs: AMBARI-21181 > https://issues.apache.org/jira/browse/AMBARI-21181 > > > Repository: ambari > > > Description > --- > > Have a new mapper called map_anonymize which accepts a pattern containing > elements. After the mapping if the pattern is found in the field, than > the parts matching the elements are replaced with a configurable > hiding character ( default 'X' ). > > Example: > pattern: the secret is / which is confidential > log message: Bla bla bla the secret is SECRET1 / SECRET2 which is > confidential bla bla > after the mapping: Bla bla bla the secret is XXX / XXX which is > confidential bla bla > > This patch also fixes the checkpoint file writing, which wasn't atomic. The > checkpoint files are now written under a temporary name first, than > atomically renamed to the proper name, thus making it impossible to have a > half written file. > > > Diffs > - > > > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapAnonymizeDescriptor.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapDateDescriptor.java > f88435f > > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapFieldDescriptor.java > db086c5 > > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapFieldValueDescriptor.java > cf37e62 > > ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/inputconfig/impl/MapAnonymizeDescriptorImpl.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/inputconfig/impl/PostMapValuesAdapter.java > 32aded8 > ambari-logsearch/ambari-logsearch-logfeeder/docs/postMapValues.md 906359a > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java > 2bc4e68 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java > fba596d > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java > 8c76785 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputS3File.java > 4bf162b > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/mapper/MapperAnonymize.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java > ffd6cec > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/alias_config.json > e2ed625 > > ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/mapper/MapperAnonymizeTest.java > PRE-CREATION > ambari-logsearch/ambari-logsearch-logfeeder/src/test/resources/log4j.xml > e641018 > > ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapDate.java > dcb0393 > > ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapField.java > b18439c > > ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapFieldAnonymize.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerPostMapValuesListDeserializer.java > 18744e2 > > ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-hst.json > 4ccef74 > > ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-zookeeper.json > d3685a4 > > > Diff: https://reviews.apache.org/r/59828/diff/1/ > > > Testing > --- > > Tested on local cluster, and by the new unit tests added. > > > Thanks, > > Miklos Gergely > >
Review Request 59828: Ability to anonymize data during log processing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59828/ --- Review request for Ambari, Oliver Szabo and Robert Nettleton. Bugs: AMBARI-21181 https://issues.apache.org/jira/browse/AMBARI-21181 Repository: ambari Description --- Have a new mapper called map_anonymize which accepts a pattern containing elements. After the mapping if the pattern is found in the field, than the parts matching the elements are replaced with a configurable hiding character ( default 'X' ). Example: pattern: the secret is / which is confidential log message: Bla bla bla the secret is SECRET1 / SECRET2 which is confidential bla bla after the mapping: Bla bla bla the secret is XXX / XXX which is confidential bla bla This patch also fixes the checkpoint file writing, which wasn't atomic. The checkpoint files are now written under a temporary name first, than atomically renamed to the proper name, thus making it impossible to have a half written file. Diffs - ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapAnonymizeDescriptor.java PRE-CREATION ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapDateDescriptor.java f88435f ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapFieldDescriptor.java db086c5 ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/inputconfig/MapFieldValueDescriptor.java cf37e62 ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/inputconfig/impl/MapAnonymizeDescriptorImpl.java PRE-CREATION ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/inputconfig/impl/PostMapValuesAdapter.java 32aded8 ambari-logsearch/ambari-logsearch-logfeeder/docs/postMapValues.md 906359a ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java 2bc4e68 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java fba596d ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java 8c76785 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputS3File.java 4bf162b ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/mapper/MapperAnonymize.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java ffd6cec ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/alias_config.json e2ed625 ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/mapper/MapperAnonymizeTest.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/test/resources/log4j.xml e641018 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapDate.java dcb0393 ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapField.java b18439c ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerMapFieldAnonymize.java PRE-CREATION ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerPostMapValuesListDeserializer.java 18744e2 ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-hst.json 4ccef74 ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-zookeeper.json d3685a4 Diff: https://reviews.apache.org/r/59828/diff/1/ Testing --- Tested on local cluster, and by the new unit tests added. Thanks, Miklos Gergely
