[
https://issues.apache.org/jira/browse/KAFKA-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774202#comment-17774202
]
James Olsen edited comment on KAFKA-14546 at 10/11/23 7:11 PM:
---------------------------------------------------------------
I'm still hoping for a solution to this issue. I can understand the reluctance
to allow a {{Partitioner}} to return {{RecordMetadata.UNKNOWN_PARTITION}}
however there is another solution that would work. As my requirement discussed
above is to ensure consistent partitioning for each Topic using a single
{{{}Partitioner{}}}, it would be suitable to have the {{Partitioner}} indicate
to the {{KafkaProducer}} if it handles the Topic. This could be implemented as
follows:
{code:java}
public interface Partitioner ... {
...
default boolean partitionsTopic(String topic) {
return true;
}
...
}
{code}
{{KafkaProducer}} would call this method first and fallback to default
partitioning if it returns {{{}false{}}}. This would be backwards compatible
with existing code and have minimal overhead.
was (Author: jim_b_o):
I'm still hoping for a solution to this issue. I can understand the reluctance
to allow a `Partitioner` to return `RecordMetadata.UNKNOWN_PARTITION` however
there is another solution that would work. As my requirement discussed above
is to ensure consistent partitioning for each Topic using a single Partitioner,
it would be suitable to have the `Partitioner` indicate to the `KafkaProducer`
if it handles the Topic. This could be implemented as follows:
{code:java}
public interface Partitioner ... {
...
default boolean partitionsTopic(String topic) {
return true;
}
...
}
{code}
`KafkaProducer` would call this method first and fallback to default
partitioning if it returns `false`. This would be backwards compatible with
existing code and have minimal overhead.
> Allow Partitioner to return -1 to indicate default partitioning
> ---------------------------------------------------------------
>
> Key: KAFKA-14546
> URL: https://issues.apache.org/jira/browse/KAFKA-14546
> Project: Kafka
> Issue Type: Improvement
> Components: producer
> Affects Versions: 3.3.1
> Reporter: James Olsen
> Priority: Major
>
> Prior to KIP-794 it was possible to create a custom Partitioner that could
> delegate to the DefaultPartitioner. DefaultPartitioner has been deprecated
> so we can now only delegate to BuiltInPartitioner.partitionForKey which does
> not handle a non-keyed message. Hence there is now no mechanism for a custom
> Partitioner to fallback to default partitioning, e.g. for the non-keyed
> sticky case.
> I would like to propose that KafkaProducer.partition(...) not throw
> IllegalArgumentException if the Partitioner returns
> RecordMetadata.UNKNOWN_PARTITION and instead continue with the default
> behaviour. Maybe with a configuration flag to enable this behaviour so as
> not to break existing expectations?
> Why was Partitioner delegation with default fallback useful?
> # A single Producer can be used to write to multiple Topics where each Topic
> may have different partitioning requirements. The Producer can only have a
> single Partitioner so the Partitioner needs to be able to switch behaviour
> based on the Topic, including the need to fallback to default behaviour if a
> given Topic does not have a custom requirement.
> # Multiple services may need to produce to the same Topic and these services
> may be authored by different teams. A single custom Partitioner that
> encapsulates all Topic specific partitioning logic can be used by all teams
> at all times for all Topics ensuring that mistakes are not made.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
