[
https://issues.apache.org/jira/browse/KAFKA-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659432#comment-16659432
]
Matthias J. Sax commented on KAFKA-7510:
----------------------------------------
Why does it not contradict KAFKA-6538? It is about adding key/value in human
readable form to exception messages that end up in the logs. While this ticket
is about removing key/value data from the logs. What do you mean by "it only
affects it's implementation"?
For all the other points: I think discussion this on the ticket is not the
right place. Would you like to drive a discussion [~MrKafka]? If yes, please
send an "[DISCUSS]" email to dev-mailing list or maybe propose a KIP (cf.
[https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals)?|https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals)]
The goal should be to agree on a global strategy on how to handle this, and
this should be documented in the Wiki IMHO to have a reference. For concrete
violations of the strategy (eg, in RecordCollectorImpl) we can create
individual ticket like this one to close the gaps.
Thoughts?
> KStreams RecordCollectorImpl leaks data to logs on error
> --------------------------------------------------------
>
> Key: KAFKA-7510
> URL: https://issues.apache.org/jira/browse/KAFKA-7510
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Mr Kafka
> Priority: Major
> Labels: user-experience
>
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl leaks data
> on error as it dumps the *value* / message payload to the logs.
> This is problematic as it may contain personally identifiable information
> (pii) or other secret information to plain text log files which can then be
> propagated to other log systems i.e Splunk.
> I suggest the *key*, and *value* fields be moved to debug level as it is
> useful for some people while error level contains the *errorMessage,
> timestamp, topic* and *stackTrace*.
> {code:java}
> private <K, V> void recordSendError(
> final K key,
> final V value,
> final Long timestamp,
> final String topic,
> final Exception exception
> ) {
> String errorLogMessage = LOG_MESSAGE;
> String errorMessage = EXCEPTION_MESSAGE;
> if (exception instanceof RetriableException) {
> errorLogMessage += PARAMETER_HINT;
> errorMessage += PARAMETER_HINT;
> }
> log.error(errorLogMessage, key, value, timestamp, topic,
> exception.toString());
> sendException = new StreamsException(
> String.format(
> errorMessage,
> logPrefix,
> "an error caught",
> key,
> value,
> timestamp,
> topic,
> exception.toString()
> ),
> exception);
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
