[
https://issues.apache.org/jira/browse/KAFKA-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661665#comment-16661665
]
ASF GitHub Bot commented on KAFKA-7510:
---------------------------------------
forficate opened a new pull request #5834: refs KAFKA-7510, preventing data
being leaked to logs by default
URL: https://github.com/apache/kafka/pull/5834
Refs KAFKA-7510 (https://issues.apache.org/jira/browse/KAFKA-7510).
This is a bare minimum implementation to get data out of the logs by default
as explained in the Jira ticket. I've tried to keep the formatting consistent
with the existing formatting.
A further follow up PR as part of the wider KAFKA-7510 discussion can add
unit tests asserting that data is not present at ERROR log level.
Note I do get flickering tests but they look to be unrelated and non
deterministic due to the parallelism although they do appear to be mostly be
from `RocksDBWindowsStoreTest` when I do get failures.
```
org.apache.kafka.streams.state.StoresTest > shouldCreateRocksDbWindowStore
PASSED
org.apache.kafka.streams.state.StoresTest >
shouldThrowIfILruMapStoreNameIsNull PASSED
org.apache.kafka.streams.state.StoresTest >
shouldThrowIfIPersistentSessionStoreStoreNameIsNull PASSED
org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest >
testRestore SKIPPED
> Task :streams:test FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':streams:test'.
> Process 'Gradle Test Executor 103' finished with non-zero exit value 134
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the user guide at
https://docs.gradle.org/4.10.2/userguide/java_plugin.html#sec:test_execution
```
Note I've also yet to find out the `gradle` task to publish the artifact to
my local maven repository to test in my app, `maven-publish` plugin doesn't
look to be enabled and not been able to dig in to the build scripts enough to
get it to work. Using `assembly` and manually adding the JAR to the classpath
to test is non trivial with my build pipeline/app deployment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> KStreams RecordCollectorImpl leaks data to logs on error
> --------------------------------------------------------
>
> Key: KAFKA-7510
> URL: https://issues.apache.org/jira/browse/KAFKA-7510
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Mr Kafka
> Priority: Major
> Labels: user-experience
>
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl leaks data
> on error as it dumps the *value* / message payload to the logs.
> This is problematic as it may contain personally identifiable information
> (pii) or other secret information to plain text log files which can then be
> propagated to other log systems i.e Splunk.
> I suggest the *key*, and *value* fields be moved to debug level as it is
> useful for some people while error level contains the *errorMessage,
> timestamp, topic* and *stackTrace*.
> {code:java}
> private <K, V> void recordSendError(
> final K key,
> final V value,
> final Long timestamp,
> final String topic,
> final Exception exception
> ) {
> String errorLogMessage = LOG_MESSAGE;
> String errorMessage = EXCEPTION_MESSAGE;
> if (exception instanceof RetriableException) {
> errorLogMessage += PARAMETER_HINT;
> errorMessage += PARAMETER_HINT;
> }
> log.error(errorLogMessage, key, value, timestamp, topic,
> exception.toString());
> sendException = new StreamsException(
> String.format(
> errorMessage,
> logPrefix,
> "an error caught",
> key,
> value,
> timestamp,
> topic,
> exception.toString()
> ),
> exception);
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
