Randall,
Say I've got a connector that needs to control topic creation. What I need
is an AdminClient s.t. my connector can do what it knows it needs to do.
This KIP doesn't address the issues that have been brought up wrt
configuration, principals, ACL etc, since I'll still need to construct my
own AdminClient.
Should such a connector ignore your proposed configuration settings? Should
it use it's own principal and it's own configuration properties? How does
my AdminClient's settings interact with your proposed settings and the
existing cluster settings?
What happens when a user specifies topic creation settings in a connector
config, but the connector then applies it's own topic creation logic? Are
the configurations silently ignored? If not, how can a connector honor your
proposed settings?
Do these settings apply to internal topics created by the framework on
bahalf of a connector, e.g. via KafkaConfigBackingStore?
When do the cluster settings get applied? Only after 3 layers of
fall-through?
I'd have the same questions if e.g. transformations could be ignored or
overridden by connectors or if there were multiple places to specify what
serde to use.
I don't see how controlling topic creation based on topic name is something
we should support across all connectors, as if it is some established
pattern or universally useful.
Ryanne
On Mon, Sep 24, 2018, 10:14 AM Randall Hauch <rha...@gmail.com> wrote:
> Hi, Ryanne. My apologies for not responding earlier, as I was on a long
> holiday.
>
> Thanks for your feedback and questions about this KIP. You've raised
> several points in the discussion so far, so let me try to address most of
> them.
>
> IIUC, one of your major concerns is that this KIP introduces a new way to
> define configurations for topics. That is true, and the whole reason is to
> simply the user experience for people using source connectors. You still
> have the freedom to manually pre-create topics before running a connector,
> or to rely upon the broker automatically creating topics for the connectors
> when those topics don't yet exist -- in both cases, don't include anything
> about topic creation in your connector configurations. In fact, when you do
> this, Connect uses the current behavior by assuming the topics exist or
> will be autocreated with the proper configurations.
>
> But for many environments, this existing approach is not enough. First, if
> you're relying upon the broker to autocreate topics, then the brokers
> single set of default topic settings must match the requirements of your
> new topics. This can be difficult when your running multiple kinds of
> connectors with differing expectations. Consider using a CDC connector that
> expects compaction, a high-volume web service connector that should not use
> compaction but expects deletion after 7 days, and another connector with
> lower volume that uses 30 day retention. Or, consider connectors that are
> producing to topics that have very different message characteristics:
> different sizes, different throughputs, different partitions, etc. The only
> way to work around this is to pre-create the topics, but this adds more
> complexity and room for errors, especially when a single instance of some
> source connectors can write to dozens (or even hundreds) of topics.
>
> Second, many operators prefer (or are required) to disable topic
> autocreation, since simple mistakes with command line tools can result in
> new topics. In this cases, users have no choice but to manually precreate
> the topics that complicates the process of running a connector and, as
> mentioned above, increases the risk that something goes wrong.
>
> Third, the reason why this KIP introduces a way for connector
> implementations to override some topic settings is because some source
> connectors have very specific requirements. When I wrote the first Debezium
> CDC connectors, many first-time users didn't precreate the topics as
> recommended in the documentation, and didn't change their brokers' default
> topic settings. Only after a few days when they tried reconsuming the full
> streams did they realize that Kafka had deleted messages older than the
> default retention period. Debezium expects / requires compacted topics, so
> all kinds of things went wrong. Connect is often one of the first ways in
> which people get introduced to Kafka, and they simply don't yet have an
> understanding of many of the details that you or I don't have to think
> twice about.
>
> You suggested that maybe Connect should just expose the Admin API. That's
> possible, but IMO it's very heavyweight and complex. The whole point of
> Connect's design is to abstract the connector developer away from most of
> the details of Kafka -- it doesn't even expose the producer and consumer
> APIs, which are much simpler. IMO it would be a mistake to require source
> connector developers to deal with the Admin API -- I even have trouble
> writing code that uses it to properly create topics, especially around
> properly handling all of the potential error conditions.
>
> You also mention that topic settings in a connector configuration might not
> reflect the actual topic's settings. This is true, especially if the topic
> already existed with different settings before the connector was run.
> However, this is also very true of the broker's default topic settings,
> which very often don't reflect the actual settings for all of the topics --
> the defaults may have been changed, or topics are created manually with
> very different settings. The only way to know the settings of a particular
> topic are to get them for that topic.
>
> The use of naming rules in the topic creation settings is intentional, and
> it allows connector users to define topic settings for topics based upon
> the names. In some cases this may require several rules to handle the
> different topics, but most of the time a single rule may be all that's
> required. I also don't agree that users will start naming topics to
> simplify their rules -- many source connectors that write to more than one
> topic often don't allow the user to specify the full name of the topics
> anyway, and when they do they often only write to one topic.
>
> I still think that the proposed KIP provides a simple way for most source
> connector users to control (via configuration) the settings of the topics
> that will be created by Connect for that connector, which works with all
> existing source connectors out of the box and does not add additional
> complexities for source connector developers.
>
> Best regards,
>
> Randall
>
>
>
> On Wed, Sep 12, 2018 at 12:22 PM Ryanne Dolan <ryannedo...@gmail.com>
> wrote:
>
> > > Rather than go though the connect framework, connectors should just
> > create their own AdminClient instance and create their own topics?
> >
> > Rather, can the framework be improved to expose an AdminClient ready to
> > use? Then connectors can use this instance without needing separate
> > identities/principals and associated configuration (which I totally
> > understand would be a nightmare). I believe that covers all the
> use-cases?
> >
> > I just don't see how the "terrible config situation" is remedied by
> adding
> > even more configuration.
> >
> > Also, I'm not sure I can conceive of a use-case in which a single
> connector
> > would need multiple default topic settings *based on the topic name*. Can
> > you give a real-world example? Is this something you've encountered, or
> are
> > you just trying for a flexible design?
> >
> > Ryanne
> >
> > On Tue, Sep 11, 2018 at 9:57 PM Gwen Shapira <g...@confluent.io> wrote:
> >
> > > Hi Ryanne,
> > >
> > > Thanks for the feedback!
> > >
> > > Can you explain a bit more what you mean by "if we allow connectors to
> > make
> > > this
> > > decision, they should have full control of the process."?
> > >
> > > I assume you mean, something like:
> > > Rather than go though the connect framework, connectors should just
> > create
> > > their own AdminClient instance and create their own topics?
> > >
> > > The problem with this approach is that connectors currently don't have
> > > their own identity (in the authentication/authorization sense). All
> > > connectors share the framework identity, if the users need to start
> > > configuring security for both the framework and connect itself, it gets
> > > messy rather quickly.
> > > We actually already do the thing I'm imagining you suggested in some
> > > connectors right now (create AdminClient and configure topics), and we
> > hope
> > > to use the new framework capability to clean-up the configuration mess
> > this
> > > has caused. I spent 4 days trying to figure out what a specific
> connector
> > > doesn't work, just to find out that you need to give it its own
> security
> > > config because it has an AdminClient so the configuration on the
> > framework
> > > isn't enough.
> > >
> > > From my experience with rather large number of customers, there are
> some
> > > companies where the topics are controlled by a central team that owns
> all
> > > the machinery to create and configure topics (sometimes via gitops,
> > > kubernetes custom resources, etc) and they would indeed be very
> surprised
> > > if a connector suddenly had opinions about topics. There are also teams
> > > where the application developers feel like they know their data and
> > > use-case the best and they are in-charge of making all topic-level
> > > decisions, usually automated by the app itself. Admin client was
> created
> > > for those teams and I think they'll appreciate having this capability
> in
> > > connect too. Funny thing is, customers who work with one model usually
> > > can't believe the other model even exists.
> > >
> > > I'd love to propose a compromise and suggest that we'll allow this
> > > functionality in Connect but also give ops teams the option to disable
> it
> > > and avoid surprises. But I'm afraid this wont work - too often the
> > defaults
> > > are just terrible for specific connectors (CDC connectors sometimes
> need
> > a
> > > single partition to maintain consistency) and if there is a chance the
> > > connector preference won't be used, connectors will have to force it
> via
> > > admin client which brings us back to the terrible config situation we
> > > currently have with Admin client.
> > >
> > > Gwen
> > >
> > >
> > > On Tue, Sep 11, 2018 at 7:23 PM, Ryanne Dolan <ryannedo...@gmail.com>
> > > wrote:
> > >
> > > > Randall,
> > > >
> > > > I have some concerns with this proposal.
> > > >
> > > > Firstly, I don't believe it is the job of a connector to configure
> > > topics,
> > > > generally, nor for topic-specific settings to hang out in connector
> > > > configurations. Automatic creation of topics with default settings is
> > an
> > > > established pattern elsewhere, and I don't think connectors need to
> > > diverge
> > > > from this.
> > > >
> > > > I agree there are cases where the default settings don't make sense
> and
> > > > it'd be nice to override them. But if we allow connectors to make
> this
> > > > decision, they should have full control of the process.
> > > >
> > > > Some concerns:
> > > > - I'd expect the cluster's default settings to apply to newly created
> > > > topics, regardless of who created them. I wouldn't expect source
> > > connectors
> > > > to be a special case. In particular, I'd be surprised if Kafka
> Connect
> > > were
> > > > to ignore my cluster's default settings and apply its own defaults.
> > > > - It will be possible to add a specific topic to this configuration,
> in
> > > > which case any reader would expect the topic to have the specified
> > > > settings. But this will not generally be true. Thus, the
> configuration
> > > will
> > > > end up lying and misleading, and there won't be any indication that
> the
> > > > configuration is lying.
> > > > - Connectors that want to control settings will end up naming topics
> > > > accordingly. For example, a connector that wants to control the
> number
> > of
> > > > partitions would need a bunch of creation rules for 1 partition, 2
> > > > partitions and so on. This is a bad pattern to establish. A better
> > > pattern
> > > > is to let the connector control the number of partitions directly
> when
> > > that
> > > > feature is required.
> > > > - The proposal introduces 2 new interfaces to control topic creation
> > > > (configuration rules and TopicSettings), where there is already a
> > > perfectly
> > > > good one (AdminClient).
> > > >
> > > > Ryanne
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Sep 4, 2018 at 5:08 PM Randall Hauch <rha...@gmail.com>
> wrote:
> > > >
> > > > > Okay, I think I cleaned up the formatting issues in the KIP wiki
> > page.
> > > > And
> > > > > while implementing I realized that it'd be helpful to be able to
> > limit
> > > > via
> > > > > the connector configuration and the rules which topics are
> created. I
> > > > added
> > > > > the `topic.creation.${ruleName}.policy` behavior, with possible
> > values
> > > > of
> > > > > "create" (the default), "autocreate" (to specify that Connect
> should
> > > let
> > > > > the broker autocreate any matching topics), and "fail" (to specify
> > that
> > > > > Connect should not allow the creation of topics whose names match
> the
> > > > > rule's regular expression).
> > > > >
> > > > > Let me know what you think. I'd like to start voting soon, but
> > because
> > > I
> > > > > made the above change I'll wait a few days.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Randall
> > > > >
> > > > > On Wed, Aug 29, 2018 at 9:41 PM Randall Hauch <rha...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Thanks, Magesh.
> > > > > >
> > > > > > All, I've made a few very minor changes to some JavaDocs and the
> > > > > > signatures of the name-value pair methods in TopicSettings
> > > interface. I
> > > > > > also described as a fifth rejected alternative why this KIP does
> > not
> > > > > modify
> > > > > > any topic settings for existing topics. All of these are pretty
> > > minor,
> > > > > but
> > > > > > I'm happy to hear about issues or suggestions.
> > > > > >
> > > > > > Since the above changes were very minor, I'll kick off a vote to
> > > accept
> > > > > > this KIP unless I hear something in the next day or two. Note
> that
> > > this
> > > > > KIP
> > > > > > has been around in virtually the exact form for over a year.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Randall
> > > > > >
> > > > > > On Wed, Aug 29, 2018 at 9:17 PM Magesh Nandakumar <
> > > > mage...@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > >> Randall,
> > > > > >>
> > > > > >> I originally thought that this proposal was a config only topic
> > > > settings
> > > > > >> and hence made the comment about configs being pass through. I
> > just
> > > > > >> realized that the connectors can also override and provide the
> > > > > >> TopicSettings. With that in mind, I think the proposal looks
> > great.
> > > > > >> Looking forward to the feature.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Magesh
> > > > > >>
> > > > > >> On Tue, Aug 28, 2018 at 8:53 PM Magesh Nandakumar <
> > > > mage...@confluent.io
> > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I was wondering if it would be much simpler to just do a
> > > > pass-through
> > > > > so
> > > > > >> > that we can support any topic setting added in Kafka without
> any
> > > > code
> > > > > >> > changes in connect. Since these are for topics that will have
> > the
> > > > > actual
> > > > > >> > data stream, users might possibly need more flexibility in
> terms
> > > of
> > > > > how
> > > > > >> the
> > > > > >> > topics get created.
> > > > > >> >
> > > > > >> > Thanks
> > > > > >> > Magesh
> > > > > >> >
> > > > > >> > On Tue, Aug 28, 2018 at 4:56 PM Randall Hauch <
> rha...@gmail.com
> > >
> > > > > wrote:
> > > > > >> >
> > > > > >> >> Do you think we should support name-value pairs, too?
> > > > > >> >>
> > > > > >> >> On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar <
> > > > > >> mage...@confluent.io>
> > > > > >> >> wrote:
> > > > > >> >>
> > > > > >> >> > Randall,
> > > > > >> >> >
> > > > > >> >> > Thanks a lot for the KIP. I think this would be a great
> > > addition
> > > > > for
> > > > > >> >> many
> > > > > >> >> > source connectors.
> > > > > >> >> > One clarification I had was regarding the topic settings
> that
> > > can
> > > > > be
> > > > > >> >> > configured. Is it limited to the setting exposed in the
> > > > > TopicSettings
> > > > > >> >> > interface?
> > > > > >> >> >
> > > > > >> >> > Thanks
> > > > > >> >> > Magesh
> > > > > >> >> >
> > > > > >> >> > On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch <
> > > rha...@gmail.com>
> > > > > >> wrote:
> > > > > >> >> >
> > > > > >> >> > > Okay, after much delay let's try this again for AK 2.1.
> Has
> > > > > anyone
> > > > > >> >> found
> > > > > >> >> > > any concerns? Stephane suggested that we allow updating
> > topic
> > > > > >> >> > > configurations (everything but partition count). I'm
> > > > unconvinced
> > > > > >> that
> > > > > >> >> > it's
> > > > > >> >> > > worth the additional complexity in the implementation and
> > the
> > > > > >> >> > documentation
> > > > > >> >> > > to explain the behavior. Changing several of the
> > > topic-specific
> > > > > >> >> > > configurations have significant impact on broker
> behavior /
> > > > > >> >> > functionality,
> > > > > >> >> > > so IMO we need to proceed more cautiously.
> > > > > >> >> > >
> > > > > >> >> > > Stephane, do you have a particular use case in mind for
> > > > updating
> > > > > >> topic
> > > > > >> >> > > configurations on an existing topic?
> > > > > >> >> > >
> > > > > >> >> > > Randall
> > > > > >> >> > >
> > > > > >> >> > >
> > > > > >> >> > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <
> > > > rha...@gmail.com>
> > > > > >> >> wrote:
> > > > > >> >> > >
> > > > > >> >> > > > The KIP deadline for 1.1 has already passed, but I'd
> like
> > > to
> > > > > >> restart
> > > > > >> >> > this
> > > > > >> >> > > > discussion so that we make the next release. I've not
> yet
> > > > > >> addressed
> > > > > >> >> the
> > > > > >> >> > > > previous comment about *existing* topics, but I'll try
> to
> > > do
> > > > > that
> > > > > >> >> over
> > > > > >> >> > > the
> > > > > >> >> > > > next few weeks. Any other
> comments/suggestions/questions?
> > > > > >> >> > > >
> > > > > >> >> > > > Best regards,
> > > > > >> >> > > >
> > > > > >> >> > > > Randall
> > > > > >> >> > > >
> > > > > >> >> > > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <
> > > > > rha...@gmail.com
> > > > > >> >
> > > > > >> >> > wrote:
> > > > > >> >> > > >
> > > > > >> >> > > >> Oops. Yes, I meant “replication factor”.
> > > > > >> >> > > >>
> > > > > >> >> > > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu <
> > yuzhih...@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >> >> > > >> >
> > > > > >> >> > > >> > Randall:
> > > > > >> >> > > >> > bq. AdminClient currently allows changing the
> > > replication
> > > > > >> >> factory.
> > > > > >> >> > > >> >
> > > > > >> >> > > >> > By 'replication factory' did you mean 'replication
> > > > factor' ?
> > > > > >> >> > > >> >
> > > > > >> >> > > >> > Cheers
> > > > > >> >> > > >> >
> > > > > >> >> > > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <
> > > > > >> rha...@gmail.com
> > > > > >> >> >
> > > > > >> >> > > >> wrote:
> > > > > >> >> > > >> >>
> > > > > >> >> > > >> >> Currently the KIP's scope is only topics that don't
> > yet
> > > > > >> exist,
> > > > > >> >> and
> > > > > >> >> > we
> > > > > >> >> > > >> have
> > > > > >> >> > > >> >> to cognizant of race conditions between tasks with
> > the
> > > > same
> > > > > >> >> > > connector.
> > > > > >> >> > > >> I
> > > > > >> >> > > >> >> think it is worthwhile to consider whether the
> KIP's
> > > > scope
> > > > > >> >> should
> > > > > >> >> > > >> expand to
> > > > > >> >> > > >> >> also address *existing* partitions, though it may
> not
> > > be
> > > > > >> >> > appropriate
> > > > > >> >> > > to
> > > > > >> >> > > >> >> have as much control when changing the topic
> settings
> > > for
> > > > > an
> > > > > >> >> > existing
> > > > > >> >> > > >> >> topic. For example, changing the number of
> partitions
> > > > > (which
> > > > > >> the
> > > > > >> >> > KIP
> > > > > >> >> > > >> >> considers a "topic-specific setting" even though
> > > > > technically
> > > > > >> it
> > > > > >> >> is
> > > > > >> >> > > not)
> > > > > >> >> > > >> >> shouldn't be done blindly due to the partitioning
> > > > impacts,
> > > > > >> and
> > > > > >> >> IIRC
> > > > > >> >> > > you
> > > > > >> >> > > >> >> can't reduce them (which we could verify before
> > > > applying).
> > > > > >> >> Also, I
> > > > > >> >> > > >> don't
> > > > > >> >> > > >> >> think the AdminClient currently allows changing the
> > > > > >> replication
> > > > > >> >> > > >> factory. I
> > > > > >> >> > > >> >> think changing the topic configs is less
> problematic
> > > both
> > > > > >> from
> > > > > >> >> what
> > > > > >> >> > > >> makes
> > > > > >> >> > > >> >> sense for connectors to verify/change and from what
> > the
> > > > > >> >> AdminClient
> > > > > >> >> > > >> >> supports.
> > > > > >> >> > > >> >>
> > > > > >> >> > > >> >> Even if we decide that it's not appropriate to
> change
> > > the
> > > > > >> >> settings
> > > > > >> >> > on
> > > > > >> >> > > >> an
> > > > > >> >> > > >> >> existing topic, I do think it's advantageous to at
> > > least
> > > > > >> notify
> > > > > >> >> the
> > > > > >> >> > > >> >> connector (or task) prior to the first record sent
> > to a
> > > > > given
> > > > > >> >> topic
> > > > > >> >> > > so
> > > > > >> >> > > >> that
> > > > > >> >> > > >> >> the connector can fail or issue a warning if it
> > doesn't
> > > > > meet
> > > > > >> its
> > > > > >> >> > > >> >> requirements.
> > > > > >> >> > > >> >>
> > > > > >> >> > > >> >> Best regards,
> > > > > >> >> > > >> >>
> > > > > >> >> > > >> >> Randall
> > > > > >> >> > > >> >>
> > > > > >> >> > > >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek <
> > > > > >> >> > > >> >> steph...@simplemachines.com.au> wrote:
> > > > > >> >> > > >> >>
> > > > > >> >> > > >> >>> Hi Randall,
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>> Thanks for the KIP. I like it
> > > > > >> >> > > >> >>> What happens when the target topic is already
> > created
> > > > but
> > > > > >> the
> > > > > >> >> > > configs
> > > > > >> >> > > >> do
> > > > > >> >> > > >> >>> not match?
> > > > > >> >> > > >> >>> i.e. wrong RF, num partitions, or missing /
> > additional
> > > > > >> configs?
> > > > > >> >> > Will
> > > > > >> >> > > >> you
> > > > > >> >> > > >> >>> attempt to apply the necessary changes or throw an
> > > > error?
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>> Thanks!
> > > > > >> >> > > >> >>> Stephane
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" <
> > > > > >> >> > > mathieu.fenn...@replicon.com
> > > > > >> >> > > >> >
> > > > > >> >> > > >> >>> wrote:
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>> Ah, yes, I see you a highlighted part that
> > > should've
> > > > > made
> > > > > >> >> this
> > > > > >> >> > > >> clear
> > > > > >> >> > > >> >>> to me the first read. :-) Much clearer now!
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>> By the way, enjoyed your Debezium talk in NYC.
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>> Looking forward to this Kafka Connect change;
> it
> > > will
> > > > > >> allow
> > > > > >> >> me
> > > > > >> >> > to
> > > > > >> >> > > >> >>> remove a post-deployment tool that I hacked
> > > together
> > > > > for
> > > > > >> the
> > > > > >> >> > > >> purpose
> > > > > >> >> > > >> >>> of ensuring auto-created topics have the right
> > > > config.
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>> Mathieu
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>> On Tue, May 23, 2017 at 11:38 AM, Randall
> Hauch <
> > > > > >> >> > > rha...@gmail.com>
> > > > > >> >> > > >> >>> wrote:
> > > > > >> >> > > >> >>>> Thanks for the quick feedback, Mathieu. Yes, the
> > > first
> > > > > >> >> > > >> >> configuration
> > > > > >> >> > > >> >>> rule
> > > > > >> >> > > >> >>>> whose regex matches will be applied, and no other
> > > rules
> > > > > >> will
> > > > > >> >> be
> > > > > >> >> > > >> >>> used. I've
> > > > > >> >> > > >> >>>> updated the KIP to try to make this more clear,
> but
> > > let
> > > > > me
> > > > > >> >> know
> > > > > >> >> > if
> > > > > >> >> > > >> >>> it's
> > > > > >> >> > > >> >>>> still not clear.
> > > > > >> >> > > >> >>>>
> > > > > >> >> > > >> >>>> Best regards,
> > > > > >> >> > > >> >>>>
> > > > > >> >> > > >> >>>> Randall
> > > > > >> >> > > >> >>>>
> > > > > >> >> > > >> >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu
> Fenniak <
> > > > > >> >> > > >> >>>> mathieu.fenn...@replicon.com> wrote:
> > > > > >> >> > > >> >>>>
> > > > > >> >> > > >> >>>>> Hi Randall,
> > > > > >> >> > > >> >>>>>
> > > > > >> >> > > >> >>>>> Awesome, very much looking forward to this.
> > > > > >> >> > > >> >>>>>
> > > > > >> >> > > >> >>>>> It isn't 100% clear from the KIP how multiple
> > > > > config-based
> > > > > >> >> rules
> > > > > >> >> > > >> >>> would
> > > > > >> >> > > >> >>>>> be applied; it looks like the first
> configuration
> > > rule
> > > > > >> whose
> > > > > >> >> > regex
> > > > > >> >> > > >> >>>>> matches the topic name will be used, and no
> other
> > > > rules
> > > > > >> will
> > > > > >> >> be
> > > > > >> >> > > >> >>>>> applied. Is that correct? (I wasn't sure if it
> > > might
> > > > > >> >> cascade
> > > > > >> >> > > >> >>>>> together multiple matching rules...)
> > > > > >> >> > > >> >>>>>
> > > > > >> >> > > >> >>>>> Looks great,
> > > > > >> >> > > >> >>>>>
> > > > > >> >> > > >> >>>>> Mathieu
> > > > > >> >> > > >> >>>>>
> > > > > >> >> > > >> >>>>>
> > > > > >> >> > > >> >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch <
> > > > > >> >> > rha...@gmail.com>
> > > > > >> >> > > >> >>> wrote:
> > > > > >> >> > > >> >>>>>> Hi, all.
> > > > > >> >> > > >> >>>>>>
> > > > > >> >> > > >> >>>>>> We recently added the ability for Kafka Connect
> > to
> > > > > create
> > > > > >> >> > > >> >>> *internal*
> > > > > >> >> > > >> >>>>> topics
> > > > > >> >> > > >> >>>>>> using the new AdminClient, but it still would
> be
> > > > great
> > > > > if
> > > > > >> >> Kafka
> > > > > >> >> > > >> >>> Connect
> > > > > >> >> > > >> >>>>>> could do this for new topics that result from
> > > source
> > > > > >> >> connector
> > > > > >> >> > > >> >>> records.
> > > > > >> >> > > >> >>>>>> I've outlined an approach to do this in
> "KIP-158
> > > > Kafka
> > > > > >> >> Connect
> > > > > >> >> > > >> >>> should
> > > > > >> >> > > >> >>>>> allow
> > > > > >> >> > > >> >>>>>> source connectors to set topic-specific
> settings
> > > for
> > > > > new
> > > > > >> >> > > >> >> topics".
> > > > > >> >> > > >> >>>>>>
> > > > > >> >> > > >> >>>>>> *
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> >> > > >> >>>>> 158%3A+Kafka+Connect+should+
> > > > allow+source+connectors+to+
> > > > > >> >> > > >> >>>>> set+topic-specific+settings+for+new+topics
> > > > > >> >> > > >> >>>>>> <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> >> > > >> >>>>> 158%3A+Kafka+Connect+should+
> > > > allow+source+connectors+to+
> > > > > >> >> > > >> >>>>> set+topic-specific+settings+for+new+topics>*
> > > > > >> >> > > >> >>>>>>
> > > > > >> >> > > >> >>>>>> Please take a look and provide feedback.
> Thanks!
> > > > > >> >> > > >> >>>>>>
> > > > > >> >> > > >> >>>>>> Best regards,
> > > > > >> >> > > >> >>>>>>
> > > > > >> >> > > >> >>>>>> Randall
> > > > > >> >> > > >> >>>>>
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>>
> > > > > >> >> > > >> >>
> > > > > >> >> > > >>
> > > > > >> >> > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Gwen Shapira*
> > > Product Manager | Confluent
> > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > > <http://www.confluent.io/blog>
> > >
> >
>
