Hi again, I am using confluent OSS, however there is nothing magical there that helps me with this problem. Let me explain the problem.
Setting up the cluster + zookeeper is not a problem, making changes to existing configurations regarding security and other configs (with cli) is a problem. In our enterprise, we use kafka for ingestion from multiple "sources" into multiple topics (hundreds or more) in our cluster. Security (authentication+authorization) is very important so we use both client certs and user+passwd. We use groups, ACLs, quotas (+ other configs) per topic and this can be quite complicated. One thing that we have decided is that EVERYTHING MUST BE IN GIT and that it must be SIMPLE. This means that if we want to make a change on the cluster we only want to run an ansible playbook and everything should be as we want. To solve this we configure everything in yaml files where we have a nice way of describing all these things. This also works fine the first time we configure the cluster since then we only have to add things. Two main problems: - The next issue is adding a single parameter, since then we have to run the complete setup again (almost) since we cannot perform a diff of what we have already created and what already exist. Problem: Takes long time. - Next thing is to delete a parameter, which means remove a line from the config. To solve deletion we need a diff and since we cannot get structured data, this currently does not work (although we currently use an error prone approach that sucks...) I am a bit surprised that more people does not have problems with this kind of things, I have found one or two pet projects on github that solves a part of this but not more. Any input is appreciated Regards /Johan mån 13 aug. 2018 kl. 14:28 skrev M. Manna <manme...@gmail.com>: > If you are working on automating your kafka setup, wouldn't it be better > for you to use confluent OSS? Or, are you aiming for something very much > home-grown? > > On 13 August 2018 at 13:25, Johan Rask <johan.r...@klipsk.se> wrote: > > > Hi, > > we are working on automating our kafka setup (topics, users, acls etc) > but > > we are missing structured output (think json) from the cli tools (i.e > > kafka-acls.sh, kafka-topics.sh). It only gives us human readable output > > which is hard to parse. > > > > We would like to be able to use it with common tools like bash scripts, > jq > > and ansible . Is there any existing solution or any ongoing initiative > that > > can help us with this? > > > > Regards /Johan > > >