Hey all, please consider voting for this KIP. It's really a shame that topic creation is impossible when clusters are under-provisioned, which is not uncommon in a dynamic environment like Kubernetes.
Ryanne On Thu, Feb 6, 2020 at 10:57 AM Mickael Maison <mickael.mai...@gmail.com> wrote: > I have not seen any new feedback nor votes. > Bumping this thread again > > On Mon, Jan 27, 2020 at 3:55 PM Mickael Maison <mickael.mai...@gmail.com> > wrote: > > > > Hi, > > > > We are now at 4 non-binding votes but still no binding votes. > > I have not seen any outstanding questions in the DISCUSS thread. If > > you have any feedback, please let me know. > > > > Thanks > > > > > > On Thu, Jan 16, 2020 at 2:03 PM M. Manna <manme...@gmail.com> wrote: > > > > > > MIckael, > > > > > > > > > > > > On Thu, 16 Jan 2020 at 14:01, Mickael Maison <mickael.mai...@gmail.com > > > > > wrote: > > > > > > > Hi Manna, > > > > > > > > In your example, the topic 'dummy' is not under replicated. It just > > > > has 1 replica. A topic under replicated is a topic with less ISRs > than > > > > replicas. > > > > > > > > Having under replicated topics is relatively common in a Kafka > > > > cluster, it happens everytime is broker is down. However Kafka does > > > > not permit it to happen at topic creation. Currently at creation, > > > > Kafka requires to have at least as many brokers as the replication > > > > factor. This KIP addresses this limitation. > > > > > > > > Regarding your 2nd point. When rack awareness is enabled, Kafka tries > > > > to distribute partitions across racks. When all brokers in a rack are > > > > down (ie: a zone is offline), you can end up with partitions not well > > > > distributed even with rack awareness. There are currently no easy way > > > > to track such partitions so I decided to not attempt addressing this > > > > issue in this KIP. > > > > > > > > I hope that answers your questions. > > > > > > > > > > It does and I appreciate you taking time and explaining this. > > > > > > +1 (binding) if I haven't already. > > > > > > > > > > > > > > > > > > > On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash > > > > <kamal.chandraprak...@gmail.com> wrote: > > > > > > > > > > +1 (non-binding). Thanks for the KIP! > > > > > > > > > > On Mon, Jan 13, 2020 at 1:58 PM M. Manna <manme...@gmail.com> > wrote: > > > > > > > > > > > Hi Mikael, > > > > > > > > > > > > Apologies for last minute question, as I just caught up with it. > > > > Thanks for > > > > > > your work on the KIP. > > > > > > > > > > > > Just trying to get your thoughts on one thing (I might have > > > > misunderstood > > > > > > it) - currently it's possible (even though I am strongly against > it) to > > > > > > create Kafka topics which are under-replicated; despite all > brokers > > > > being > > > > > > online. This the the output of an intentionally under-replicated > topic > > > > > > "dummy" with p=6 and RF=1 (with a 3 node cluster) > > > > > > > > > > > > > > > > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ > ./kafka-topics.sh > > > > > > --create --topic dummy --partitions 6 --replication-factor 1 > > > > > > --bootstrap-server localhost:9092 > > > > > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ > ./kafka-topics.sh > > > > > > --describe --topic dummy --bootstrap-server localhost:9092 > > > > > > Topic:dummy PartitionCount:6 ReplicationFactor:1 > > > > > > > > > > > > > > > > > Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520 > > > > > > Topic: dummy Partition: 0 Leader: 3 > Replicas: 3 > > > > > > Isr: 3 > > > > > > Topic: dummy Partition: 1 Leader: 1 > Replicas: 1 > > > > > > Isr: 1 > > > > > > Topic: dummy Partition: 2 Leader: 2 > Replicas: 2 > > > > > > Isr: 2 > > > > > > Topic: dummy Partition: 3 Leader: 3 > Replicas: 3 > > > > > > Isr: 3 > > > > > > Topic: dummy Partition: 4 Leader: 1 > Replicas: 1 > > > > > > Isr: 1 > > > > > > Topic: dummy Partition: 5 Leader: 2 > Replicas: 2 > > > > > > Isr: 2 > > > > > > > > > > > > This is with respect to the following statement on your KIP > (i.e. > > > > > > under-replicated topic creation is also permitted when none is > > > > offline): > > > > > > > > > > > > *but note that this may already happen (without this KIP) when > > > > > > > topics/partitions are created while all brokers in a rack are > offline > > > > > > (ie: > > > > > > > an availability zone is offline). Tracking topics/partitions > not > > > > > > optimally > > > > > > > spread across all racks can be tackled in a follow up KIP. * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Did you mean to say that such under-replicated topics (including > > > > > > human-created ones) will be handled in a separete KIP ? > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > On Mon, 13 Jan 2020 at 10:15, Mickael Maison < > mickael.mai...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi all. > > > > > > > > > > > > > > With 2.5.0 approaching, bumping this thread once more as > feedback or > > > > > > > votes would be nice. > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley < > tbent...@redhat.com> > > > > wrote: > > > > > > > > > > > > > > > > +1 non-binding. Thanks! > > > > > > > > > > > > > > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau > > > > > > > > <soenke.lie...@opencore.com.invalid> wrote: > > > > > > > > > > > > > > > > > Hi Mickael, > > > > > > > > > > > > > > > > > > thanks for your response! That all makes perfect sense and > I > > > > cannot > > > > > > > > > give any actual use cases for where what I asked about > would be > > > > > > useful > > > > > > > > > :) > > > > > > > > > It was more the idle thought if this might be low hanging > fruit > > > > while > > > > > > > > > changing this anyway to avoid having to circle back later > on and > > > > > > > > > wanted to at least mention it. > > > > > > > > > > > > > > > > > > I am totally happy either way! > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > Sönke > > > > > > > > > > > > > > > > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison < > > > > > > mickael.mai...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Thanks Sönke for the feedback. > > > > > > > > > > > > > > > > > > > > I debated this point quite a bit before deciding to base > > > > creation > > > > > > > > > > around "min.insync.replicas". > > > > > > > > > > > > > > > > > > > > For me, the goal of this KIP is to enable administrators > to > > > > provide > > > > > > > > > > higher availability. In a 3 node cluster configured for > high > > > > > > > > > > availability (3 replicas, 2 min ISR), by enabling this > feature, > > > > > > > > > > clusters should be fully usable even when 1 broker is > down. > > > > This > > > > > > > > > > should cover all "normal" maintenance operations like a > rolling > > > > > > > > > > restart or just the recovery of a broker. > > > > > > > > > > > > > > > > > > > > At the moment, when creating a topic/partition, the > assumption > > > > is > > > > > > > that > > > > > > > > > > the resource will be fully functioning. This KIP does not > > > > change > > > > > > this > > > > > > > > > > assumption. If this is something someone wants, I think > it > > > > should > > > > > > be > > > > > > > > > > handled in a different KIP that targets that use case. By > > > > relying > > > > > > on > > > > > > > > > > "min.insync.replicas", we don't break any assumptions > the user > > > > has > > > > > > > and > > > > > > > > > > this should be fully transparent from the user point of > view. > > > > > > > > > > > > > > > > > > > > About "min.insync.replicas", one caveat that is not > explicit > > > > in the > > > > > > > > > > KIP is that it's currently possible to create topics > with less > > > > > > > > > > replicas than this settings. For that reason, I think the > > > > > > > > > > implementation will actually rely on min(replicas, > min-isr) > > > > instead > > > > > > > of > > > > > > > > > > simply min.insync.replicas. I have updated the KIP to > > > > explicitly > > > > > > > > > > mention this point. > > > > > > > > > > > > > > > > > > > > I hope that answers your question, let me know. > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau > > > > > > > > > > <soenke.lie...@opencore.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > > Hi Michael, > > > > > > > > > > > > > > > > > > > > > > that sounds like a useful addition! I can't help but > wonder > > > > > > > whether by > > > > > > > > > > > leaving in the restriction that "min.insync.replicas" > has to > > > > be > > > > > > > > > > > satisfied we'll be back here in a years time because > someone > > > > has > > > > > > a > > > > > > > > > > > scenario where he or she wants to go below that :) > > > > > > > > > > > I don't have a strong opinion either way to be honest, > just a > > > > > > > random > > > > > > > > > > > thought when reading the KIP. > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > Sönke > > > > > > > > > > > > > > > > > > > > > > On Thu, 12 Dec 2019 at 22:44, Ryanne Dolan < > > > > > > ryannedo...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > +1 non-binding, thx > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Dec 12, 2019 at 6:09 AM Mickael Maison < > > > > > > > > > mickael.mai...@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Bumping this thread, I've not seen any votes or > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Nov 13, 2019 at 12:17 PM Mickael Maison > > > > > > > > > > > > > <mickael.mai...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to start a vote on KIP-409: Allow > creating > > > > > > > > > under-replicated > > > > > > > > > > > > > > topics and partitions > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Sönke Liebau > > > > > > > > > > > Partner > > > > > > > > > > > Tel. +49 179 7940878 > > > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 > Wedel - > > > > > > > Germany > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Sönke Liebau > > > > > > > > > Partner > > > > > > > > > Tel. +49 179 7940878 > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 > Wedel - > > > > Germany > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
