On Mon, Jul 24, 2017 at 1:50 PM, Florian Hussonnois <fhussonn...@gmail.com> wrote:
> Hi all, > > Here is two suggestions for the Kafka Connect API that I prefer to present > you first before creating KIPs. > > 1/ > Currently, in Kafka Connect we distinguish two configuration types: the > worgker config and the connector config. The connector config is accessible > from the Connector class through th method "*void start(Map<String, String> > props)*" but developers can't access the originals worker configuration > (i.e bootstrap.servers, etc). > > I think some use cases could require an internal producer or consumer > created from the Connector/Task instance. > > For example in order to : > - To consume a topic for special events which will trigger tasks > reconfiguration > - To produce messages in order to track tasks progression (the messages > could be consumed by a KafkaStreams app in order to expose monitoring UI). > > Currently, the only way to do this is to define the kafka configuration in > each configuration but this is not really developer friendly. > I think we'd want to be careful about doing this. If you run connect like a service, you may need to grant the service relatively broad permissions in Kafka to support many users. But you still may want to lock down what any given user of the service can do. We currently don't have ACLs (or even authenticated REST API access), but I know that there are requests for this functionality. If we just exposed the entire worker config to connectors, they'd be able to do anything the worker has permission for. By the way, for your second use case, there's no reason you can't do that today. Just emit more producer records but point them at a different topic and > > 2/ > Secondly, it could be useful to have a method returning the connector name > accessible from the Connector class. For example, the connector name could > be used as an identifier for retrieving sensitives settings from a > configuration management server (for example Consul). > > Do you think this two minor suggestions can deserved KIPs ? > This is small and seems reasonable. You could also, of course, just save the configuration or that specific property when connector.start(config) is invoked. A more general extension would be to have ConnectorContext expose the entire original config instead of having a method specifically for the name. -Ewen > > Thanks very much. > > Florian. > > > -- > Florian HUSSONNOIS >
