FWIW, I’d find this feature useful. On Mon, Sep 24, 2018 at 2:42 PM Randall Hauch <rha...@gmail.com> wrote:
> Ryanne, > > If your connector is already using the AdminClient, then you as the > developer have a choice of switching to the new Connect-based functionality > or keeping the existing use of the AdminClient. If the connector uses both > mechanisms (which I wouldn't recommend, simply because of the complexity of > it for a user), then the topic will be created by the first mechanism to > actually attempt and successfully create the topic(s) in the Kafka cluster > that the Connect worker uses. As mentioned in the KIP, "This feature ... > does not change the topic-specific settings on any existing topics." IOW, > if the topic already exists, it can't be created again and therefore the > `topic.creation.*` properties will not apply for that existing topic. > > > Do these settings apply to internal topics created by the framework on > > bahalf of a connector, e.g. via KafkaConfigBackingStore? > > No, they don't, and I'm happy to add a clarification to the KIP if you feel > it is necessary. > > > I'd have the same questions if e.g. transformations could be ignored or > > overridden by connectors or if there were multiple places to specify what > > serde to use. > > There are multiple places that converters can be defined: the worker config > defines the key and value converters that will be used for all connectors, > except when a connector defines its own key and value converters. > > > I don't see how controlling topic creation based on topic name is > something > > we should support across all connectors, as if it is some established > > pattern or universally useful. > > Topics are identified by name, and when you create a topic with specific > settings or change a topic's settings you identify the topic by name. The > fact that this KIP uses regular expressions to match topic names doesn't > seem surprising, since we use regexes elsewhere. > > Best regards > > On Mon, Sep 24, 2018 at 1:24 PM Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > Randall, > > > > Say I've got a connector that needs to control topic creation. What I > need > > is an AdminClient s.t. my connector can do what it knows it needs to do. > > This KIP doesn't address the issues that have been brought up wrt > > configuration, principals, ACL etc, since I'll still need to construct my > > own AdminClient. > > > > Should such a connector ignore your proposed configuration settings? > Should > > it use it's own principal and it's own configuration properties? How does > > my AdminClient's settings interact with your proposed settings and the > > existing cluster settings? > > > > What happens when a user specifies topic creation settings in a connector > > config, but the connector then applies it's own topic creation logic? Are > > the configurations silently ignored? If not, how can a connector honor > your > > proposed settings? > > > > Do these settings apply to internal topics created by the framework on > > bahalf of a connector, e.g. via KafkaConfigBackingStore? > > > > When do the cluster settings get applied? Only after 3 layers of > > fall-through? > > > > I'd have the same questions if e.g. transformations could be ignored or > > overridden by connectors or if there were multiple places to specify what > > serde to use. > > > > I don't see how controlling topic creation based on topic name is > something > > we should support across all connectors, as if it is some established > > pattern or universally useful. > > > > Ryanne > > > > On Mon, Sep 24, 2018, 10:14 AM Randall Hauch <rha...@gmail.com> wrote: > > > > > Hi, Ryanne. My apologies for not responding earlier, as I was on a long > > > holiday. > > > > > > Thanks for your feedback and questions about this KIP. You've raised > > > several points in the discussion so far, so let me try to address most > of > > > them. > > > > > > IIUC, one of your major concerns is that this KIP introduces a new way > to > > > define configurations for topics. That is true, and the whole reason is > > to > > > simply the user experience for people using source connectors. You > still > > > have the freedom to manually pre-create topics before running a > > connector, > > > or to rely upon the broker automatically creating topics for the > > connectors > > > when those topics don't yet exist -- in both cases, don't include > > anything > > > about topic creation in your connector configurations. In fact, when > you > > do > > > this, Connect uses the current behavior by assuming the topics exist or > > > will be autocreated with the proper configurations. > > > > > > But for many environments, this existing approach is not enough. First, > > if > > > you're relying upon the broker to autocreate topics, then the brokers > > > single set of default topic settings must match the requirements of > your > > > new topics. This can be difficult when your running multiple kinds of > > > connectors with differing expectations. Consider using a CDC connector > > that > > > expects compaction, a high-volume web service connector that should not > > use > > > compaction but expects deletion after 7 days, and another connector > with > > > lower volume that uses 30 day retention. Or, consider connectors that > are > > > producing to topics that have very different message characteristics: > > > different sizes, different throughputs, different partitions, etc. The > > only > > > way to work around this is to pre-create the topics, but this adds more > > > complexity and room for errors, especially when a single instance of > some > > > source connectors can write to dozens (or even hundreds) of topics. > > > > > > Second, many operators prefer (or are required) to disable topic > > > autocreation, since simple mistakes with command line tools can result > in > > > new topics. In this cases, users have no choice but to manually > precreate > > > the topics that complicates the process of running a connector and, as > > > mentioned above, increases the risk that something goes wrong. > > > > > > Third, the reason why this KIP introduces a way for connector > > > implementations to override some topic settings is because some source > > > connectors have very specific requirements. When I wrote the first > > Debezium > > > CDC connectors, many first-time users didn't precreate the topics as > > > recommended in the documentation, and didn't change their brokers' > > default > > > topic settings. Only after a few days when they tried reconsuming the > > full > > > streams did they realize that Kafka had deleted messages older than the > > > default retention period. Debezium expects / requires compacted topics, > > so > > > all kinds of things went wrong. Connect is often one of the first ways > in > > > which people get introduced to Kafka, and they simply don't yet have an > > > understanding of many of the details that you or I don't have to think > > > twice about. > > > > > > You suggested that maybe Connect should just expose the Admin API. > That's > > > possible, but IMO it's very heavyweight and complex. The whole point of > > > Connect's design is to abstract the connector developer away from most > of > > > the details of Kafka -- it doesn't even expose the producer and > consumer > > > APIs, which are much simpler. IMO it would be a mistake to require > source > > > connector developers to deal with the Admin API -- I even have trouble > > > writing code that uses it to properly create topics, especially around > > > properly handling all of the potential error conditions. > > > > > > You also mention that topic settings in a connector configuration might > > not > > > reflect the actual topic's settings. This is true, especially if the > > topic > > > already existed with different settings before the connector was run. > > > However, this is also very true of the broker's default topic settings, > > > which very often don't reflect the actual settings for all of the > topics > > -- > > > the defaults may have been changed, or topics are created manually with > > > very different settings. The only way to know the settings of a > > particular > > > topic are to get them for that topic. > > > > > > The use of naming rules in the topic creation settings is intentional, > > and > > > it allows connector users to define topic settings for topics based > upon > > > the names. In some cases this may require several rules to handle the > > > different topics, but most of the time a single rule may be all that's > > > required. I also don't agree that users will start naming topics to > > > simplify their rules -- many source connectors that write to more than > > one > > > topic often don't allow the user to specify the full name of the topics > > > anyway, and when they do they often only write to one topic. > > > > > > I still think that the proposed KIP provides a simple way for most > source > > > connector users to control (via configuration) the settings of the > topics > > > that will be created by Connect for that connector, which works with > all > > > existing source connectors out of the box and does not add additional > > > complexities for source connector developers. > > > > > > Best regards, > > > > > > Randall > > > > > > > > > > > > On Wed, Sep 12, 2018 at 12:22 PM Ryanne Dolan <ryannedo...@gmail.com> > > > wrote: > > > > > > > > Rather than go though the connect framework, connectors should just > > > > create their own AdminClient instance and create their own topics? > > > > > > > > Rather, can the framework be improved to expose an AdminClient ready > to > > > > use? Then connectors can use this instance without needing separate > > > > identities/principals and associated configuration (which I totally > > > > understand would be a nightmare). I believe that covers all the > > > use-cases? > > > > > > > > I just don't see how the "terrible config situation" is remedied by > > > adding > > > > even more configuration. > > > > > > > > Also, I'm not sure I can conceive of a use-case in which a single > > > connector > > > > would need multiple default topic settings *based on the topic name*. > > Can > > > > you give a real-world example? Is this something you've encountered, > or > > > are > > > > you just trying for a flexible design? > > > > > > > > Ryanne > > > > > > > > On Tue, Sep 11, 2018 at 9:57 PM Gwen Shapira <g...@confluent.io> > > wrote: > > > > > > > > > Hi Ryanne, > > > > > > > > > > Thanks for the feedback! > > > > > > > > > > Can you explain a bit more what you mean by "if we allow connectors > > to > > > > make > > > > > this > > > > > decision, they should have full control of the process."? > > > > > > > > > > I assume you mean, something like: > > > > > Rather than go though the connect framework, connectors should just > > > > create > > > > > their own AdminClient instance and create their own topics? > > > > > > > > > > The problem with this approach is that connectors currently don't > > have > > > > > their own identity (in the authentication/authorization sense). All > > > > > connectors share the framework identity, if the users need to start > > > > > configuring security for both the framework and connect itself, it > > gets > > > > > messy rather quickly. > > > > > We actually already do the thing I'm imagining you suggested in > some > > > > > connectors right now (create AdminClient and configure topics), and > > we > > > > hope > > > > > to use the new framework capability to clean-up the configuration > > mess > > > > this > > > > > has caused. I spent 4 days trying to figure out what a specific > > > connector > > > > > doesn't work, just to find out that you need to give it its own > > > security > > > > > config because it has an AdminClient so the configuration on the > > > > framework > > > > > isn't enough. > > > > > > > > > > From my experience with rather large number of customers, there are > > > some > > > > > companies where the topics are controlled by a central team that > owns > > > all > > > > > the machinery to create and configure topics (sometimes via gitops, > > > > > kubernetes custom resources, etc) and they would indeed be very > > > surprised > > > > > if a connector suddenly had opinions about topics. There are also > > teams > > > > > where the application developers feel like they know their data and > > > > > use-case the best and they are in-charge of making all topic-level > > > > > decisions, usually automated by the app itself. Admin client was > > > created > > > > > for those teams and I think they'll appreciate having this > capability > > > in > > > > > connect too. Funny thing is, customers who work with one model > > usually > > > > > can't believe the other model even exists. > > > > > > > > > > I'd love to propose a compromise and suggest that we'll allow this > > > > > functionality in Connect but also give ops teams the option to > > disable > > > it > > > > > and avoid surprises. But I'm afraid this wont work - too often the > > > > defaults > > > > > are just terrible for specific connectors (CDC connectors sometimes > > > need > > > > a > > > > > single partition to maintain consistency) and if there is a chance > > the > > > > > connector preference won't be used, connectors will have to force > it > > > via > > > > > admin client which brings us back to the terrible config situation > we > > > > > currently have with Admin client. > > > > > > > > > > Gwen > > > > > > > > > > > > > > > On Tue, Sep 11, 2018 at 7:23 PM, Ryanne Dolan < > ryannedo...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > Randall, > > > > > > > > > > > > I have some concerns with this proposal. > > > > > > > > > > > > Firstly, I don't believe it is the job of a connector to > configure > > > > > topics, > > > > > > generally, nor for topic-specific settings to hang out in > connector > > > > > > configurations. Automatic creation of topics with default > settings > > is > > > > an > > > > > > established pattern elsewhere, and I don't think connectors need > to > > > > > diverge > > > > > > from this. > > > > > > > > > > > > I agree there are cases where the default settings don't make > sense > > > and > > > > > > it'd be nice to override them. But if we allow connectors to make > > > this > > > > > > decision, they should have full control of the process. > > > > > > > > > > > > Some concerns: > > > > > > - I'd expect the cluster's default settings to apply to newly > > created > > > > > > topics, regardless of who created them. I wouldn't expect source > > > > > connectors > > > > > > to be a special case. In particular, I'd be surprised if Kafka > > > Connect > > > > > were > > > > > > to ignore my cluster's default settings and apply its own > defaults. > > > > > > - It will be possible to add a specific topic to this > > configuration, > > > in > > > > > > which case any reader would expect the topic to have the > specified > > > > > > settings. But this will not generally be true. Thus, the > > > configuration > > > > > will > > > > > > end up lying and misleading, and there won't be any indication > that > > > the > > > > > > configuration is lying. > > > > > > - Connectors that want to control settings will end up naming > > topics > > > > > > accordingly. For example, a connector that wants to control the > > > number > > > > of > > > > > > partitions would need a bunch of creation rules for 1 partition, > 2 > > > > > > partitions and so on. This is a bad pattern to establish. A > better > > > > > pattern > > > > > > is to let the connector control the number of partitions directly > > > when > > > > > that > > > > > > feature is required. > > > > > > - The proposal introduces 2 new interfaces to control topic > > creation > > > > > > (configuration rules and TopicSettings), where there is already a > > > > > perfectly > > > > > > good one (AdminClient). > > > > > > > > > > > > Ryanne > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 4, 2018 at 5:08 PM Randall Hauch <rha...@gmail.com> > > > wrote: > > > > > > > > > > > > > Okay, I think I cleaned up the formatting issues in the KIP > wiki > > > > page. > > > > > > And > > > > > > > while implementing I realized that it'd be helpful to be able > to > > > > limit > > > > > > via > > > > > > > the connector configuration and the rules which topics are > > > created. I > > > > > > added > > > > > > > the `topic.creation.${ruleName}.policy` behavior, with possible > > > > values > > > > > > of > > > > > > > "create" (the default), "autocreate" (to specify that Connect > > > should > > > > > let > > > > > > > the broker autocreate any matching topics), and "fail" (to > > specify > > > > that > > > > > > > Connect should not allow the creation of topics whose names > match > > > the > > > > > > > rule's regular expression). > > > > > > > > > > > > > > Let me know what you think. I'd like to start voting soon, but > > > > because > > > > > I > > > > > > > made the above change I'll wait a few days. > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Randall > > > > > > > > > > > > > > On Wed, Aug 29, 2018 at 9:41 PM Randall Hauch < > rha...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Thanks, Magesh. > > > > > > > > > > > > > > > > All, I've made a few very minor changes to some JavaDocs and > > the > > > > > > > > signatures of the name-value pair methods in TopicSettings > > > > > interface. I > > > > > > > > also described as a fifth rejected alternative why this KIP > > does > > > > not > > > > > > > modify > > > > > > > > any topic settings for existing topics. All of these are > pretty > > > > > minor, > > > > > > > but > > > > > > > > I'm happy to hear about issues or suggestions. > > > > > > > > > > > > > > > > Since the above changes were very minor, I'll kick off a vote > > to > > > > > accept > > > > > > > > this KIP unless I hear something in the next day or two. Note > > > that > > > > > this > > > > > > > KIP > > > > > > > > has been around in virtually the exact form for over a year. > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > Randall > > > > > > > > > > > > > > > > On Wed, Aug 29, 2018 at 9:17 PM Magesh Nandakumar < > > > > > > mage...@confluent.io> > > > > > > > > wrote: > > > > > > > > > > > > > > > >> Randall, > > > > > > > >> > > > > > > > >> I originally thought that this proposal was a config only > > topic > > > > > > settings > > > > > > > >> and hence made the comment about configs being pass > through. I > > > > just > > > > > > > >> realized that the connectors can also override and provide > the > > > > > > > >> TopicSettings. With that in mind, I think the proposal looks > > > > great. > > > > > > > >> Looking forward to the feature. > > > > > > > >> > > > > > > > >> Thanks, > > > > > > > >> Magesh > > > > > > > >> > > > > > > > >> On Tue, Aug 28, 2018 at 8:53 PM Magesh Nandakumar < > > > > > > mage...@confluent.io > > > > > > > > > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> > I was wondering if it would be much simpler to just do a > > > > > > pass-through > > > > > > > so > > > > > > > >> > that we can support any topic setting added in Kafka > without > > > any > > > > > > code > > > > > > > >> > changes in connect. Since these are for topics that will > > have > > > > the > > > > > > > actual > > > > > > > >> > data stream, users might possibly need more flexibility in > > > terms > > > > > of > > > > > > > how > > > > > > > >> the > > > > > > > >> > topics get created. > > > > > > > >> > > > > > > > > >> > Thanks > > > > > > > >> > Magesh > > > > > > > >> > > > > > > > > >> > On Tue, Aug 28, 2018 at 4:56 PM Randall Hauch < > > > rha...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > >> > > > > > > > > >> >> Do you think we should support name-value pairs, too? > > > > > > > >> >> > > > > > > > >> >> On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar < > > > > > > > >> mage...@confluent.io> > > > > > > > >> >> wrote: > > > > > > > >> >> > > > > > > > >> >> > Randall, > > > > > > > >> >> > > > > > > > > >> >> > Thanks a lot for the KIP. I think this would be a great > > > > > addition > > > > > > > for > > > > > > > >> >> many > > > > > > > >> >> > source connectors. > > > > > > > >> >> > One clarification I had was regarding the topic > settings > > > that > > > > > can > > > > > > > be > > > > > > > >> >> > configured. Is it limited to the setting exposed in the > > > > > > > TopicSettings > > > > > > > >> >> > interface? > > > > > > > >> >> > > > > > > > > >> >> > Thanks > > > > > > > >> >> > Magesh > > > > > > > >> >> > > > > > > > > >> >> > On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch < > > > > > rha...@gmail.com> > > > > > > > >> wrote: > > > > > > > >> >> > > > > > > > > >> >> > > Okay, after much delay let's try this again for AK > 2.1. > > > Has > > > > > > > anyone > > > > > > > >> >> found > > > > > > > >> >> > > any concerns? Stephane suggested that we allow > updating > > > > topic > > > > > > > >> >> > > configurations (everything but partition count). I'm > > > > > > unconvinced > > > > > > > >> that > > > > > > > >> >> > it's > > > > > > > >> >> > > worth the additional complexity in the implementation > > and > > > > the > > > > > > > >> >> > documentation > > > > > > > >> >> > > to explain the behavior. Changing several of the > > > > > topic-specific > > > > > > > >> >> > > configurations have significant impact on broker > > > behavior / > > > > > > > >> >> > functionality, > > > > > > > >> >> > > so IMO we need to proceed more cautiously. > > > > > > > >> >> > > > > > > > > > >> >> > > Stephane, do you have a particular use case in mind > for > > > > > > updating > > > > > > > >> topic > > > > > > > >> >> > > configurations on an existing topic? > > > > > > > >> >> > > > > > > > > > >> >> > > Randall > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch < > > > > > > rha...@gmail.com> > > > > > > > >> >> wrote: > > > > > > > >> >> > > > > > > > > > >> >> > > > The KIP deadline for 1.1 has already passed, but > I'd > > > like > > > > > to > > > > > > > >> restart > > > > > > > >> >> > this > > > > > > > >> >> > > > discussion so that we make the next release. I've > not > > > yet > > > > > > > >> addressed > > > > > > > >> >> the > > > > > > > >> >> > > > previous comment about *existing* topics, but I'll > > try > > > to > > > > > do > > > > > > > that > > > > > > > >> >> over > > > > > > > >> >> > > the > > > > > > > >> >> > > > next few weeks. Any other > > > comments/suggestions/questions? > > > > > > > >> >> > > > > > > > > > > >> >> > > > Best regards, > > > > > > > >> >> > > > > > > > > > > >> >> > > > Randall > > > > > > > >> >> > > > > > > > > > > >> >> > > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch < > > > > > > > rha...@gmail.com > > > > > > > >> > > > > > > > > >> >> > wrote: > > > > > > > >> >> > > > > > > > > > > >> >> > > >> Oops. Yes, I meant “replication factor”. > > > > > > > >> >> > > >> > > > > > > > >> >> > > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu < > > > > yuzhih...@gmail.com > > > > > > > > > > > > > >> wrote: > > > > > > > >> >> > > >> > > > > > > > > >> >> > > >> > Randall: > > > > > > > >> >> > > >> > bq. AdminClient currently allows changing the > > > > > replication > > > > > > > >> >> factory. > > > > > > > >> >> > > >> > > > > > > > > >> >> > > >> > By 'replication factory' did you mean > 'replication > > > > > > factor' ? > > > > > > > >> >> > > >> > > > > > > > > >> >> > > >> > Cheers > > > > > > > >> >> > > >> > > > > > > > > >> >> > > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch < > > > > > > > >> rha...@gmail.com > > > > > > > >> >> > > > > > > > > >> >> > > >> wrote: > > > > > > > >> >> > > >> >> > > > > > > > >> >> > > >> >> Currently the KIP's scope is only topics that > > don't > > > > yet > > > > > > > >> exist, > > > > > > > >> >> and > > > > > > > >> >> > we > > > > > > > >> >> > > >> have > > > > > > > >> >> > > >> >> to cognizant of race conditions between tasks > > with > > > > the > > > > > > same > > > > > > > >> >> > > connector. > > > > > > > >> >> > > >> I > > > > > > > >> >> > > >> >> think it is worthwhile to consider whether the > > > KIP's > > > > > > scope > > > > > > > >> >> should > > > > > > > >> >> > > >> expand to > > > > > > > >> >> > > >> >> also address *existing* partitions, though it > may > > > not > > > > > be > > > > > > > >> >> > appropriate > > > > > > > >> >> > > to > > > > > > > >> >> > > >> >> have as much control when changing the topic > > > settings > > > > > for > > > > > > > an > > > > > > > >> >> > existing > > > > > > > >> >> > > >> >> topic. For example, changing the number of > > > partitions > > > > > > > (which > > > > > > > >> the > > > > > > > >> >> > KIP > > > > > > > >> >> > > >> >> considers a "topic-specific setting" even > though > > > > > > > technically > > > > > > > >> it > > > > > > > >> >> is > > > > > > > >> >> > > not) > > > > > > > >> >> > > >> >> shouldn't be done blindly due to the > partitioning > > > > > > impacts, > > > > > > > >> and > > > > > > > >> >> IIRC > > > > > > > >> >> > > you > > > > > > > >> >> > > >> >> can't reduce them (which we could verify before > > > > > > applying). > > > > > > > >> >> Also, I > > > > > > > >> >> > > >> don't > > > > > > > >> >> > > >> >> think the AdminClient currently allows changing > > the > > > > > > > >> replication > > > > > > > >> >> > > >> factory. I > > > > > > > >> >> > > >> >> think changing the topic configs is less > > > problematic > > > > > both > > > > > > > >> from > > > > > > > >> >> what > > > > > > > >> >> > > >> makes > > > > > > > >> >> > > >> >> sense for connectors to verify/change and from > > what > > > > the > > > > > > > >> >> AdminClient > > > > > > > >> >> > > >> >> supports. > > > > > > > >> >> > > >> >> > > > > > > > >> >> > > >> >> Even if we decide that it's not appropriate to > > > change > > > > > the > > > > > > > >> >> settings > > > > > > > >> >> > on > > > > > > > >> >> > > >> an > > > > > > > >> >> > > >> >> existing topic, I do think it's advantageous to > > at > > > > > least > > > > > > > >> notify > > > > > > > >> >> the > > > > > > > >> >> > > >> >> connector (or task) prior to the first record > > sent > > > > to a > > > > > > > given > > > > > > > >> >> topic > > > > > > > >> >> > > so > > > > > > > >> >> > > >> that > > > > > > > >> >> > > >> >> the connector can fail or issue a warning if it > > > > doesn't > > > > > > > meet > > > > > > > >> its > > > > > > > >> >> > > >> >> requirements. > > > > > > > >> >> > > >> >> > > > > > > > >> >> > > >> >> Best regards, > > > > > > > >> >> > > >> >> > > > > > > > >> >> > > >> >> Randall > > > > > > > >> >> > > >> >> > > > > > > > >> >> > > >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane > Maarek > > < > > > > > > > >> >> > > >> >> steph...@simplemachines.com.au> wrote: > > > > > > > >> >> > > >> >> > > > > > > > >> >> > > >> >>> Hi Randall, > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> Thanks for the KIP. I like it > > > > > > > >> >> > > >> >>> What happens when the target topic is already > > > > created > > > > > > but > > > > > > > >> the > > > > > > > >> >> > > configs > > > > > > > >> >> > > >> do > > > > > > > >> >> > > >> >>> not match? > > > > > > > >> >> > > >> >>> i.e. wrong RF, num partitions, or missing / > > > > additional > > > > > > > >> configs? > > > > > > > >> >> > Will > > > > > > > >> >> > > >> you > > > > > > > >> >> > > >> >>> attempt to apply the necessary changes or > throw > > an > > > > > > error? > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> Thanks! > > > > > > > >> >> > > >> >>> Stephane > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" < > > > > > > > >> >> > > mathieu.fenn...@replicon.com > > > > > > > >> >> > > >> > > > > > > > > >> >> > > >> >>> wrote: > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> Ah, yes, I see you a highlighted part that > > > > > should've > > > > > > > made > > > > > > > >> >> this > > > > > > > >> >> > > >> clear > > > > > > > >> >> > > >> >>> to me the first read. :-) Much clearer > now! > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> By the way, enjoyed your Debezium talk in > > NYC. > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> Looking forward to this Kafka Connect > change; > > > it > > > > > will > > > > > > > >> allow > > > > > > > >> >> me > > > > > > > >> >> > to > > > > > > > >> >> > > >> >>> remove a post-deployment tool that I hacked > > > > > together > > > > > > > for > > > > > > > >> the > > > > > > > >> >> > > >> purpose > > > > > > > >> >> > > >> >>> of ensuring auto-created topics have the > > right > > > > > > config. > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> Mathieu > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> On Tue, May 23, 2017 at 11:38 AM, Randall > > > Hauch < > > > > > > > >> >> > > rha...@gmail.com> > > > > > > > >> >> > > >> >>> wrote: > > > > > > > >> >> > > >> >>>> Thanks for the quick feedback, Mathieu. Yes, > > the > > > > > first > > > > > > > >> >> > > >> >> configuration > > > > > > > >> >> > > >> >>> rule > > > > > > > >> >> > > >> >>>> whose regex matches will be applied, and no > > other > > > > > rules > > > > > > > >> will > > > > > > > >> >> be > > > > > > > >> >> > > >> >>> used. I've > > > > > > > >> >> > > >> >>>> updated the KIP to try to make this more > clear, > > > but > > > > > let > > > > > > > me > > > > > > > >> >> know > > > > > > > >> >> > if > > > > > > > >> >> > > >> >>> it's > > > > > > > >> >> > > >> >>>> still not clear. > > > > > > > >> >> > > >> >>>> > > > > > > > >> >> > > >> >>>> Best regards, > > > > > > > >> >> > > >> >>>> > > > > > > > >> >> > > >> >>>> Randall > > > > > > > >> >> > > >> >>>> > > > > > > > >> >> > > >> >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu > > > Fenniak < > > > > > > > >> >> > > >> >>>> mathieu.fenn...@replicon.com> wrote: > > > > > > > >> >> > > >> >>>> > > > > > > > >> >> > > >> >>>>> Hi Randall, > > > > > > > >> >> > > >> >>>>> > > > > > > > >> >> > > >> >>>>> Awesome, very much looking forward to this. > > > > > > > >> >> > > >> >>>>> > > > > > > > >> >> > > >> >>>>> It isn't 100% clear from the KIP how > multiple > > > > > > > config-based > > > > > > > >> >> rules > > > > > > > >> >> > > >> >>> would > > > > > > > >> >> > > >> >>>>> be applied; it looks like the first > > > configuration > > > > > rule > > > > > > > >> whose > > > > > > > >> >> > regex > > > > > > > >> >> > > >> >>>>> matches the topic name will be used, and no > > > other > > > > > > rules > > > > > > > >> will > > > > > > > >> >> be > > > > > > > >> >> > > >> >>>>> applied. Is that correct? (I wasn't sure > if > > it > > > > > might > > > > > > > >> >> cascade > > > > > > > >> >> > > >> >>>>> together multiple matching rules...) > > > > > > > >> >> > > >> >>>>> > > > > > > > >> >> > > >> >>>>> Looks great, > > > > > > > >> >> > > >> >>>>> > > > > > > > >> >> > > >> >>>>> Mathieu > > > > > > > >> >> > > >> >>>>> > > > > > > > >> >> > > >> >>>>> > > > > > > > >> >> > > >> >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall > > Hauch < > > > > > > > >> >> > rha...@gmail.com> > > > > > > > >> >> > > >> >>> wrote: > > > > > > > >> >> > > >> >>>>>> Hi, all. > > > > > > > >> >> > > >> >>>>>> > > > > > > > >> >> > > >> >>>>>> We recently added the ability for Kafka > > Connect > > > > to > > > > > > > create > > > > > > > >> >> > > >> >>> *internal* > > > > > > > >> >> > > >> >>>>> topics > > > > > > > >> >> > > >> >>>>>> using the new AdminClient, but it still > would > > > be > > > > > > great > > > > > > > if > > > > > > > >> >> Kafka > > > > > > > >> >> > > >> >>> Connect > > > > > > > >> >> > > >> >>>>>> could do this for new topics that result > from > > > > > source > > > > > > > >> >> connector > > > > > > > >> >> > > >> >>> records. > > > > > > > >> >> > > >> >>>>>> I've outlined an approach to do this in > > > "KIP-158 > > > > > > Kafka > > > > > > > >> >> Connect > > > > > > > >> >> > > >> >>> should > > > > > > > >> >> > > >> >>>>> allow > > > > > > > >> >> > > >> >>>>>> source connectors to set topic-specific > > > settings > > > > > for > > > > > > > new > > > > > > > >> >> > > >> >> topics". > > > > > > > >> >> > > >> >>>>>> > > > > > > > >> >> > > >> >>>>>> * > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > > >> >> > > >> >>>>> 158%3A+Kafka+Connect+should+ > > > > > > allow+source+connectors+to+ > > > > > > > >> >> > > >> >>>>> set+topic-specific+settings+for+new+topics > > > > > > > >> >> > > >> >>>>>> < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > > >> >> > > >> >>>>> 158%3A+Kafka+Connect+should+ > > > > > > allow+source+connectors+to+ > > > > > > > >> >> > > >> >>>>> set+topic-specific+settings+for+new+topics>* > > > > > > > >> >> > > >> >>>>>> > > > > > > > >> >> > > >> >>>>>> Please take a look and provide feedback. > > > Thanks! > > > > > > > >> >> > > >> >>>>>> > > > > > > > >> >> > > >> >>>>>> Best regards, > > > > > > > >> >> > > >> >>>>>> > > > > > > > >> >> > > >> >>>>>> Randall > > > > > > > >> >> > > >> >>>>> > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >>> > > > > > > > >> >> > > >> >> > > > > > > > >> >> > > >> > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *Gwen Shapira* > > > > > Product Manager | Confluent > > > > > 650.450.2760 <(650)%20450-2760> | @gwenshap > > > > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > > > > > <http://www.confluent.io/blog> > > > > > > > > > > > > > > >