i have been playing with kafka connect in standalone and distributed mode.

i like standalone because:
* i get to configure it using a file. this is easy for automated deployment
(chef, puppet, etc.). configuration using a rest api i find inconvenient.
* erors show up in log files instead of having to retrieve them using a
rest api. same argument as previous bullet point really. i know how to
automate log monitoring. rest api isnt great for this.
* isolation of connector classes. every connector has its own jvm. no jar
dependency hell.

i like distributed because:
* well its fault tolerant and can distribute workload

so this makes me wonder... how hard would it be to get the
"connect-standalone" setup where each connector has its own service(s),
configuration is done using files, and errors are written to logs, yet at
the same time i can spin up multiple services for a connector and they form
a group? and while we are at it also remove the rest api entirely, since i
dont need it, it poses a security risk, and it makes it hard to spin up
multiple connectors on same box. with such a setup i could simply deploy as
many services as i need for a connector, using either chef, or perhaps
slider on yarn, or whatever framework i need.

this is related to KAFKA-3815
<https://issues.apache.org/jira/browse/KAFKA-3815> which makes similar
arguments for container deployments

Reply via email to