[
https://issues.apache.org/jira/browse/KAFKA-15912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790672#comment-17790672
]
Greg Harris commented on KAFKA-15912:
-------------------------------------
Hey [~mimaison] thanks for the ticket. I've only thought briefly about this and
haven't found any obvious blockers, but there are some design restrictions:
# Since the javadocs for Transformation and Predicate don't mention
thread-safety, I think we have to assume that they are not thread-safe
# There is room in the API for a Transformation to be stateful and
order-sensitive, (such as packing records together) so I think we would be
unable to instantiate multiple copies of a single transform stage, and all
records would have to pass serially through a stage.
If Transformations and Predicates could declare themselves thread-safe, then we
would be able to do some finer-grained parallelism, or fallback to actor-style
parallelism (a single thread with message queue input).
I think it would be ineffective/undesirable for Transformations to take this
performance optimization burden upon themselves completely like the Task
implementations do, so we should certainly improve the framework in this area.
> Parallelize conversion and transformation steps in Connect
> ----------------------------------------------------------
>
> Key: KAFKA-15912
> URL: https://issues.apache.org/jira/browse/KAFKA-15912
> Project: Kafka
> Issue Type: Improvement
> Components: connect
> Reporter: Mickael Maison
> Priority: Major
>
> In busy Connect pipelines, the conversion and transformation steps can
> sometimes have a very significant impact on performance. This is especially
> true with large records with complex schemas, for example with CDC connectors
> like Debezium.
> Today in order to always preserve ordering, converters and transformations
> are called on one record at a time in a single thread in the Connect worker.
> As Connect usually handles records in batches (up to max.poll.records in sink
> pipelines, for source pipelines while it really depends on the connector,
> most connectors I've seen still tend to return multiple records each loop),
> it could be highly beneficial to attempt running the converters and
> transformation chain in parallel by a pool a processing threads.
> It should be possible to do some of these steps in parallel and still keep
> exact ordering. I'm even considering whether an option to lose ordering but
> allow even faster processing would make sense.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
