[
https://issues.apache.org/jira/browse/KAFKA-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch updated KAFKA-2649:
---------------------------------
Description:
The only way for Processor implementations to control partitioning of forwarded
messages is to set the partitioner class as property
{{ProducerConfig.PARTITIONER_CLASS_CONFIG}} in the StreamsConfig, which should
be set to the name of a {{org.apache.kafka.clients.producer.Partitioner}}
implementation. However, doing this requires the partitioner knowing how to
properly partition *all* topics, not just the one or few topics used by the
Processor.
Instead, Kafka Streams should make it easy to optionally add a partitioning
function for each sink used in a topology. Each sink represents a single output
topic, and thus is far simpler to implement. Additionally, the sink is already
typed with the key and value types (via serdes for the keys and values), so the
partitioner can be also be typed with the key and value types. Finally, this
also keeps the logic of choosing partitioning strategies where it belongs, as
part of building the topology.
was:
The only way for Processor implementations to control partitioning of forwarded
messages is to set the partitioner class as property "{{
ProducerConfig.PARTITIONER_CLASS_CONFIG}}" in the StreamsConfig, which should
be set to the name of a {{org.apache.kafka.clients.producer.Partitioner}}
implementation. However, doing this requires the partitioner knowing how to
properly partition *all* topics, not just the one or few topics used by the
Processor.
Instead, Kafka Streams should make it easy to optionally add a partitioning
function for each sink used in a topology. Each sink represents a single output
topic, and thus is far simpler to implement. Additionally, the sink is already
typed with the key and value types (via serdes for the keys and values), so the
partitioner can be also be typed with the key and value types. Finally, this
also keeps the logic of choosing partitioning strategies where it belongs, as
part of building the topology.
> Add support for custom partitioner in sink nodes
> ------------------------------------------------
>
> Key: KAFKA-2649
> URL: https://issues.apache.org/jira/browse/KAFKA-2649
> Project: Kafka
> Issue Type: Sub-task
> Components: kafka streams
> Reporter: Randall Hauch
> Assignee: Randall Hauch
> Fix For: 0.9.0.0
>
>
> The only way for Processor implementations to control partitioning of
> forwarded messages is to set the partitioner class as property
> {{ProducerConfig.PARTITIONER_CLASS_CONFIG}} in the StreamsConfig, which
> should be set to the name of a
> {{org.apache.kafka.clients.producer.Partitioner}} implementation. However,
> doing this requires the partitioner knowing how to properly partition *all*
> topics, not just the one or few topics used by the Processor.
> Instead, Kafka Streams should make it easy to optionally add a partitioning
> function for each sink used in a topology. Each sink represents a single
> output topic, and thus is far simpler to implement. Additionally, the sink is
> already typed with the key and value types (via serdes for the keys and
> values), so the partitioner can be also be typed with the key and value
> types. Finally, this also keeps the logic of choosing partitioning strategies
> where it belongs, as part of building the topology.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
