Dan Burkert has submitted this change and it was merged. Change subject: KUDU-1812. Redact pretty-printed sensitive user data ......................................................................
KUDU-1812. Redact pretty-printed sensitive user data This commit introduces a framework in logging.h to enable sensitive user data to be redacted in the Kudu server, and to a lesser extent, in the C++ client. - The 'log_redact_user_data' gflag is added which controls whether row and predicate data is redacted from log, error and status messages. The flag defaults to true in the Kudu server binaries order to provide a safe default for Kudu users. The flag is explicitly set to false in tests and the 'kudu' CLI tool in order to aid debugging. - There is a new thread-local boolean which indicates whether ToString functions should redact user data in the current thread. This defaults to 'true', but it is always also combined with the 'log_redact_user_data' flag to determine whether redaction will actually happen. - A utility macro KUDU_DISABLE_REDACTION(...) and RAII equivalent ScopedDisableRedaction can disable redaction while evaluating a particular expression or scope, useful in contexts such as the web UI or tools where we don't want to redact. - A macro KUDU_REDACT(expr) replaces its argument with '<redacted>' if redaction is enabled. - ToString and equivalent calls have been changed to consult log_redact_user_data and the TLS flag to determine whether to perform redaction, where appropriate. Some ToString calls specifically disable redaction based on the type of data being stringified; this behavior has been explicitly called out in the header documentation, but this isn't expected to be a source of security issues in the future since in all such cases the data is not considered sensitive. A handful of ToString methods in our public API disable redaction in order to retain the same behavior; these methods should never be used for internal Kudu logging without manual redaction, and their doxygen has been updated to indicate this. - Redaction in the C++ client uses the same mechanism as in the server, but the 'log_redact_user_data' is permanently set to 'true'. We can add a public API option to allow applications to turn off redaction in the future if it proves to be necessary. The advantage of a TLS flag in addition to 'log_redact_user_data' is that we're now able to apply KUDU_REDACT() at a very low level (the stringification functions in types.cc which are used for stringifying all user data). This means that we are by-default redacted anywhere that stringifies a row, rather than having to look for all cases that may lead to this stringification. Instead, we only have to find the places that explicitly want to disable redaction, which should be the exception rather than the rule. Redaction of raw buffers and protobuf messages will be included in a follow up commit. Change-Id: I2b87a3065280116bb8af6f26f072dafdfd1ee077 Reviewed-on: http://gerrit.cloudera.org:8080/5555 Tested-by: Kudu Jenkins Reviewed-by: Adar Dembo <a...@cloudera.com> Reviewed-on: http://gerrit.cloudera.org:8080/5565 Reviewed-by: Dan Burkert <danburk...@apache.org> --- M src/kudu/client/batcher.cc M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/meta_cache.h M src/kudu/client/scanner-internal.h M src/kudu/client/session-internal.cc M src/kudu/client/write_op.h M src/kudu/common/column_predicate-test.cc M src/kudu/common/encoded_key-test.cc M src/kudu/common/partial_row.cc M src/kudu/common/partial_row.h M src/kudu/common/partition-test.cc M src/kudu/common/partition.cc M src/kudu/common/partition.h M src/kudu/common/partition_pruner-test.cc M src/kudu/common/partition_pruner.h M src/kudu/common/row_operations.cc M src/kudu/common/row_operations.h M src/kudu/common/scan_spec-test.cc M src/kudu/common/schema-test.cc M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/integration-tests/external_mini_cluster.cc M src/kudu/server/webserver.cc M src/kudu/tablet/memrowset-test.cc M src/kudu/tablet/transactions/write_transaction.cc M src/kudu/tools/tool_action_local_replica.cc M src/kudu/tools/tool_main.cc M src/kudu/util/flag_tags-test.cc M src/kudu/util/flags-test.cc M src/kudu/util/logging-test.cc M src/kudu/util/logging.cc M src/kudu/util/logging.h M src/kudu/util/test_util.cc 34 files changed, 349 insertions(+), 80 deletions(-) Approvals: Dan Burkert: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/5565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I2b87a3065280116bb8af6f26f072dafdfd1ee077 Gerrit-PatchSet: 4 Gerrit-Project: kudu Gerrit-Branch: branch-1.2.x Gerrit-Owner: Dan Burkert <danburk...@apache.org> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Dan Burkert <danburk...@apache.org> Gerrit-Reviewer: Kudu Jenkins
