[
https://issues.apache.org/jira/browse/KAFKA-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17251851#comment-17251851
]
Randall Hauch commented on KAFKA-10865:
---------------------------------------
We currently trace-log the record passed into a transformation, but it looks
like we don't trace-log the record just before it is passed to the connector.
We may want to have slightly different log messages for source and sink
connectors. If so, that would make it difficult to add all of the log messages
in `TransformationChain`. So we might also need to think about the trace-log
messages in `WorkerSourceTask` and `WorkerSinkTask` near where the
transformation chain is called.
We need to be careful about changing existing log messages, in case users are
relying upon specific log messages for any automated tooling. Trace messages
are probably a bit easier to change if necessary, since it's unlikely those
would be enabled in production, but we should still strive for minimally
changing existing log messages.
[~govi20], thank you for volunteering to take this ticket. You assigned
yourself, so +1. Looking forward to a PR. :-)
> Improve trace-logging for Transformations (including Predicates)
> ----------------------------------------------------------------
>
> Key: KAFKA-10865
> URL: https://issues.apache.org/jira/browse/KAFKA-10865
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Robin Moffatt
> Assignee: Govinda Sakhare
> Priority: Major
> Labels: newbie
>
> I've been spending [a bunch of time poking around
> SMTs|https://rmoff.net/categories/twelvedaysofsmt/] recently, and one common
> challenge I've had is being able to debug when things don't behave as I
> expect.
>
> I know that there is the {{TransformationChain}} logger, but this only gives
> (IIUC) the input record
> {code:java}
> [2020-12-17 09:38:58,057] TRACE [sink-simulator-day12-00|task-0] Applying
> transformation io.confluent.connect.transforms.Filter$Value to
> SinkRecord{kafkaOffset=10551, timestampType=CreateTime}
> ConnectRecord{topic='day12-sys01', kafkaPartition=0,
> key=2c2ceb9b-8b31-4ade-a757-886ebfb7a398, keySchema=Schema{STRING},
> value=Struct{units=16,product=Founders Breakfast
> Stout,amount=30.41,txn_date=Sat Dec 12 18:21:18 GMT 2020,source=SYS01},
> valueSchema=Schema{io.mdrogalis.Gen0:STRUCT}, timestamp=1608197938054,
> headers=ConnectHeaders(headers=)}
> (org.apache.kafka.connect.runtime.TransformationChain:47)
> {code}
>
> I think it would be really useful to also have trace level logging that
> included:
> - the _output_ of *each* transform
> - the evaluation and result of any `predicate`s
> I have been using
> {{com.github.jcustenborder.kafka.connect.simulator.SimulatorSinkConnector}}
> which is really useful for seeing the final record:
> {code:java}
> [2020-12-17 09:38:58,057] INFO [sink-simulator-day12-00|task-0]
> record.value=Struct{units=16,product=Founders Breakfast
> Stout,amount=30.41,txn_date=Sat Dec 12 18:21:18 GMT 2020,source=SYS01}
> (com.github.jcustenborder.kafka.connect.simulator.SimulatorSinkTask:50)
> {code}
>
> But doesn't include things like topic name (which is often changed by common
> SMTs)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
