James Olsen created KAFKA-14546:
-----------------------------------
Summary: Allow Partitioner to return -1 to indicate default
partitioning
Key: KAFKA-14546
URL: https://issues.apache.org/jira/browse/KAFKA-14546
Project: Kafka
Issue Type: Improvement
Components: producer
Affects Versions: 3.3.1
Reporter: James Olsen
Prior to KIP-794 it was possible to create a custom Partitioner that could
delegate to the DefaultPartitioner. DefaultPartitioner has been deprecated so
we can now only delegate to BuiltInPartitioner.partitionForKey which does not
handle a non-keyed message. Hence there is now no mechanism for a custom
Partitioner to fallback to default partitioning, e.g. for the non-keyed sticky
case.
I would like to propose that KafkaProducer.partition(...) not throw
IllegalArgumentException if the Partitioner returns
RecordMetadata.UNKNOWN_PARTITION and instead continue with the default
behaviour. Maybe with a configuration flag to enable this behaviour so as not
to break existing expectations?
Why was Partitioner delegation with default fallback useful?
# A single Producer can be used to write to multiple Topics where each Topic
may have different partitioning requirements. The Producer can only have a
single Partitioner so the Partitioner needs to be able to switch behaviour
based on the Topic, including the need to fallback to default behaviour if a
given Topic does not have a custom requirement.
# Multiple services may need to produce to the same Topic and these services
may be authored by different teams. A single custom Partitioner that
encapsulates all Topic specific partitioning logic can be used by all teams at
all times for all Topics ensuring that mistakes are not made.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
