KIP-794 inhibits Partitioner delegation with no fallback option

[James Olsen]

Prior to KIP-794 it was possible to create a custom Partitioner that could 
delegate to the DefaultPartitioner.  This has been deprecated so we can now 
only delegate to BuiltInPartitioner.partitionForKey which does not handle a 
non-keyed message.  Hence there is now no mechanism for a custom Partitioner to 
fallback to default partitioning, e.g. for the non-keyed sticky case.

Why was Partitioner delegation useful?
1. A single Producer can be used to write to multiple Topics and each Topic may 
have different partitioning requirements.  The Producer can only have a single 
Partitioner so the Partitioner needs to be able to switch behaviour based on 
the Topic, including the need to fallback to default behaviour if a given Topic 
does not have a custom requirement.
2. Multiple services may need to produce to the same Topic and these services 
may be authored by different teams.  A single custom Partitioner that 
encapsulates the Topic specific partitioning logic can be used by all teams at 
all times for all Topics ensuring that mistakes are not made.

It looks like there would be an easy solution which does not rely on 
Partitioner delegation.  This would be to allow the Partitioner to return -1 
(RecordMetadata.UNKNOWN_PARTITION) and have the Producer fallback to default 
behaviour.  This option was rejected as the solution to KIP-794 
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner#KIP794:StrictlyUniformStickyPartitioner-Partitioner.partitiontoreturn-1)
 and dismissed as not useful to Partitioners in general.  So although not 
necessary for KIP-794 which could remain implemented as it currently is, I 
think this fallback scenario is something that should be supported - it is 
useful to custom Partitioners for the reasons outlined above.

So I would like to propose that KafkaProducer.partition(...) not throw 
IllegalArgumentException if the Partitioner returns 
RecordMetadata.UNKNOWN_PARTITION and instead continue with the default 
behaviour.  Maybe with a configuration flag to enable this behaviour so as not 
to break existing expectations?

[VOTE] 3.3.2 RC0

[Chris Egerton]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.3.2.

This is a bugfix release with several fixes since the release of 3.3.1. A
few of the major issues include:

* KAFKA-14358 Users should not be able to create a regular topic name
__cluster_metadata
KAFKA-14379 Consumer should refresh preferred read replica on update
metadata
* KAFKA-13586 Prevent exception thrown during connector update from
crashing distributed herder


Release notes for the 3.3.2 release:
https://home.apache.org/~cegerton/kafka-3.3.2-rc0/RELEASE_NOTES.html



*** Please download, test and vote by Tuesday, December 20, 10pm UTC

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~cegerton/kafka-3.3.2-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~cegerton/kafka-3.3.2-rc0/javadoc/

* Tag to be voted upon (off 3.3 branch) is the 3.3.2 tag:
https://github.com/apache/kafka/releases/tag/3.3.2-rc0

* Documentation:
https://kafka.apache.org/33/documentation.html

* Protocol:
https://kafka.apache.org/33/protocol.html

The most recent build has had test failures. These all appear to be due to
flakiness, but it would be nice if someone more familiar with the failed
tests could confirm this. I may update this thread with passing build links
if I can get one, or start a new release vote thread if test failures must
be addressed beyond re-running builds until they pass.

Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.3/135/testReport/

System tests:
http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1670984851--apache--3.3--22af3f29ce/2022-12-13--001./2022-12-13--001./report.html
(initial with three flaky failures)
Follow-up system tests:
https://home.apache.org/~cegerton/system_tests/2022-12-14--015/report.html,
https://home.apache.org/~cegerton/system_tests/2022-12-14--016/report.html,
http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1671061000--apache--3.3--69fbaf2457/2022-12-14--001./2022-12-14--001./report.html

(Note that the exact commit used for some of the system test runs will not
precisely match the commit for the release candidate, but that all
differences between those two commits should have no effect on the
relevance or accuracy of the test results.)

Thanks,

Chris

Kafka's scalability

[Péter Sinóros-Szabó]

Hi,

We have a Kafka cluster that's growing a lot year to year and I was
wondering how much it can handle without custom changes.

If you're running a large Kafka cluster or if you're aware of one, please
share some data around it:
- how many brokers is in the cluster and CPUs / RAM used
- how many topics / partitions it has
- number of messages incoming
- number of consumers / producers it supports

This could help us to see how large we can grow our clusters.

Thanks,
Peter

Need clarity on a statement in the Upgrade path

[Swathi Mocharla]

Hello,
In the upgrade docs, it is mentioned "Restart the brokers one by one for
the new protocol version to take effect. *Once the brokers begin using the
latest protocol version, it will no longer be possible to downgrade the
cluster to an older version.*"

What does this mean? Is this statement applicable to every version other
than 2.1 as well? Does that mean after the update of the inter broker
protocol, one should not rollback to a prior version?

Thanks,
Swathi

Is there an option to ignore the apiVersion between Kafka versions

[Swathi Mocharla]

Hello,
We are trying to upgrade between 2 versions of kafka where the apiVersions
of multiple apis have changed. During upgrade we notice a number of
UnsupportedVersionExceptions which we want to ignore (during upgrade).
Few questions:

   1. Is it ok to ignore mismatch of apiVersions?
   2. Is there a way to ignore them via configurations? I see "ignorable"
   tags at multiple field levels within the apis.
   3. What exactly do flexibleVersions in an api mean? for example, in the
   Fetch api, I am moving from version 12 to 13 and still running into this
   issue, while  flexibleVersions are 12+

Thanks,
Swathi