Hi Rhys, I think this will be a great addition.
Are there any performance benchmarks against Mirror Maker available? I'm interested to know if this is more performant / scalable. Regarding the implementation, here's some feedback: - I think it's worth mentioning that this solution does not rely on consumer groups, and therefore tracking progress may be tricky. Can you think of a way to expose that? - Some code can be in config Validator I believe: https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47 - I think your kip mentions `source.admin.` and `source.consumer.` but I don't see it reflected yet in the code - Is there a way to be flexible and merge list and regex, or offer the two simultaneously ? source_topics=my_static_topic,prefix.* ? Hope that helps Stephane Kind regards, Stephane [image: Simple Machines] Stephane Maarek | Developer +61 416 575 980 steph...@simplemachines.com.au simplemachines.com.au Level 2, 145 William Street, Sydney NSW 2010 On 5 June 2018 at 09:04, McCaig, Rhys <rhys_mcc...@comcast.com> wrote: > Hi All, > > As I didn’t get any comment on this KIP and there has since been an > additional 2 KIP’s created numbered 308 since, I'm bumping this and > renaming the KIP to 310 to remove the duplication: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 310%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect > > Let me know if you have any comments or feedback, would love to hear them. > > Cheers, > Rhys > > > On May 28, 2018, at 10:23 PM, McCaig, Rhys <rhys_mcc...@comcast.com> > wrote: > > > > Sorry for the bad link to the KIP, here it is: https://cwiki.apache.org/ > confluence/display/KAFKA/KIP-308%3A+Add+a+Kafka+Source+ > Connector+to+Kafka+Connect > > > >> On May 28, 2018, at 10:19 PM, McCaig, Rhys <rhys_mcc...@comcast.com> > wrote: > >> > >> Hi All, > >> > >> I added a KIP to include a Kafka Source Connector with Kafka Connect. > >> Here is the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 308%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect<htt > ps://cwiki.apache.org/confluence/display/KAFKA/KIP- > 308:+Add+a+Kafka+Source+Connector+to+Kafka+Connect> > >> > >> Looking forward to your feedback and suggestions. > >> > >> Cheers, > >> Rhys > >> > >> > > > >
