Yes, that's the approach I'm suggesting and that is mentioned in the KIP. I also propose that the distributed configuration provided in the examples set the replication factor to one but include a relevant comment.
On Mon, May 8, 2017 at 11:14 PM, BigData dev <bigdatadev...@gmail.com> wrote: > So, when Kafka broker is less than 3, and the user has not set the > replication configuration it will throw an error to the user, to correct > the configuration according to his setup? Is this the approach you are > suggesting here? > > > > On Mon, May 8, 2017 at 7:13 PM, Randall Hauch <rha...@gmail.com> wrote: > > > One of the "Rejected Alternatives" was to do something "smarter" by > > automatically reducing the replication factor when the cluster size is > > smaller than the replication factor. However, this is extremely > > unintuitive, and in rare cases (e.g., during a partial outage) might even > > result in internal topics being created with too small of a replication > > factor. And defaulting to 1 is certainly bad for production use cases, so > > that's not an option, either. > > > > While defaulting to 3 and failing if the cluster doesn't have 3 nodes is > a > > bit harsher than I'd like, it does appear to be the safer option: an > error > > message (with instructions on how to correct) is better than > inadvertently > > setting the replication factor too small and not knowing about it until > it > > is too late. > > > > On Mon, May 8, 2017 at 6:12 PM, BigData dev <bigdatadev...@gmail.com> > > wrote: > > > > > Hi, > > > I liked the KIP, as it will avoid so many errors which user can make > > during > > > setup. > > > I have 1 questions here. > > > 1. As default replication factor is set to 3, but if Kafka cluster is > > setup > > > for one node, then the user needs to override the default configuraion, > > > till then topics will not be created. > > > So, this is the behavior we want to give? > > > > > > On Mon, May 8, 2017 at 2:25 PM, Konstantine Karantasis < > > > konstant...@confluent.io> wrote: > > > > > > > Thanks a lot for the KIP Randall. This improvement should simplify > both > > > > regular deployments and testing! > > > > > > > > A minor comment. Maybe it would be nice to add a note about why > there's > > > no > > > > need for the property: config.storage.partitions > > > > I'm mentioning this for the sake of completeness, in case someone > > notices > > > > this slight asymmetry with respect to the newly introduced config > > > > properties. > > > > > > > > This is by no means a blocking comment. > > > > > > > > Thanks, > > > > Konstantine > > > > > > > > On Fri, May 5, 2017 at 7:18 PM, Randall Hauch <rha...@gmail.com> > > wrote: > > > > > > > > > Thanks, Gwen. > > > > > > > > > > Switching to low-priority is a great idea. > > > > > > > > > > The default value for the replication factor configuration is 3, > > since > > > > > that makes sense and is safe for production. Using the default > values > > > in > > > > > the example would mean it could only be run against a Kafka cluster > > > with > > > > a > > > > > minimum of 3 nodes. I propose overriding the example's replication > > > factor > > > > > configurations to be 1 so that the examples could be run on any > sized > > > > > cluster. > > > > > > > > > > The rejected alternatives mentions why the implementation doesn't > try > > > to > > > > > be too smart by calculating the replication factor. > > > > > > > > > > Best regards, > > > > > > > > > > Randall > > > > > > > > > > > On May 5, 2017, at 8:02 PM, Gwen Shapira <g...@confluent.io> > > wrote: > > > > > > > > > > > > Looks great to me :) > > > > > > > > > > > > Just one note - configurations have levels (which reflect in the > > > docs) > > > > - > > > > > I > > > > > > suggest putting the whole thing as LOW. Most users will never > need > > to > > > > > worry > > > > > > about these. For same reason I recommend leaving them out of the > > > > example > > > > > > config files - we already have issues with users playing with > > configs > > > > > > without understanding what they are doing and not liking the > > results. > > > > > > > > > > > >> On Fri, May 5, 2017 at 3:42 PM, Randall Hauch <rha...@gmail.com > > > > > > wrote: > > > > > >> > > > > > >> Hi, all. > > > > > >> > > > > > >> I've been working on KAFKA-4667 to change the distributed worker > > of > > > > > Kafka > > > > > >> Connect to look for the topics used to store connector and task > > > > > >> configurations, offsets, and status, and if those tasks do not > > exist > > > > to > > > > > >> create them using the new AdminClient. To make this as useful as > > > > > possible > > > > > >> and to minimize the need to still manually create the topics, I > > > > propose > > > > > >> adding several new distributed worker configurations to specify > > the > > > > > >> partitions and replication factor for these topics, and have > > > outlined > > > > > them > > > > > >> in "KIP-154 Add Kafka Connect configuration properties for > > creating > > > > > >> internal topics". > > > > > >> > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > >> 154+Add+Kafka+Connect+configuration+properties+for+ > > > > > >> creating+internal+topics > > > > > >> > > > > > >> Please take a look and provide feedback. Thanks! > > > > > >> > > > > > >> Best regards, > > > > > >> > > > > > >> Randall > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > *Gwen Shapira* > > > > > > Product Manager | Confluent > > > > > > 650.450.2760 | @gwenshap > > > > > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > > > > > > <http://www.confluent.io/blog> > > > > > > > > > > > > > > >
