Re: Source Connector Task in a distributed env

Thank you Ryann & Hans. I will look into it. The spooldir, I explored it too and found that it too suits for standalone as you mentioned. 'Venkata On Wed 24 Apr, 2019, 22:34 Hans Jespersen, wrote: > Your connector sounds a lot like this one > https://github.com/jcustenborder/kafka-connect-spool

Re: Source Connector Task in a distributed env

Your connector sounds a lot like this one https://github.com/jcustenborder/kafka-connect-spooldir I do not think you can run such a connector in distributed mode though. Typically something like this runs in standalone mode to avoid conflicts. -hans On Wed, Apr 24, 2019 at 1:08 AM Venkata S A

Re: Source Connector Task in a distributed env

Venkata, the example you have linked creates a single task config s.t. there is no parallelism -- a single task runs on the cluster, regardless of the number of nodes. In order to introduce parallelism, your SourceConnector needs to group all known files among N partitions and return N task configs

Source Connector Task in a distributed env

Hello Team, I am developing a custom Source Connector that watches a given directory for any new files. My question is in a Distributed environment, how will the tasks in different nodes handle the file Queue? Referring to this sample