I have a question: what if we just use UUID? That would be easy to implement. Unless there are other reasons because the id should be incremental numbers. I would like to try to get rid of this dependency because I really want to update Kafka maybe using RedPanda or Kafka Kraft...
-- Michele Sciabarra mich...@sciabarra.com ----- Original message ----- From: Brendan Doyle <bdoyle0...@gmail.com> To: dev@openwhisk.apache.org Subject: Re: Updating Kafka getting rid of Zookeeper Date: Sunday, December 18, 2022 4:36 PM Yes this is one of the things I had in my list on slack of what I would like to see accomplished in 2023 since the only zookeeper use is storing the invoker id mappings. I think it should be doable, but the one tricky part is the synchronization that the zookeeper counter provides. Whatever new mechanism that is used, it must be guaranteed that two new invokers starting at the same time wouldn’t have a race condition where they read the current counter value and assign themselves the same id. Or just the way invoker ids are represented could be changed entirely as well to be more accommodating to auto scaling up and down too; since part of the reason for the original incrementing integer invoker ids was the hashing algorithm to schedule activations to invokers which now no longer exists. I think an invoker node also no longer has to have a persisted id that is reused on restart because there’s no longer an invoker activation kafka topic with the invoker id where there may be activations left to still be processed after restart. But of all the things work wise I have on my list, this is probably the easiest value proposition to remove an entire dependency from they system. Brendan On Sun, Dec 18, 2022 at 8:36 AM Michele Sciabarra <mich...@sciabarra.com> wrote: > Hello, > > I was trying to run OpenWhisk with a newer version of Kafka and also with > RedPanda. And I failed. > I found there is a dependency on version 4 of curator that in turns blocks > zookeper to version 3.4 and newer version of Kafka uses 3.5 with a > different protocol. > > In the code, the curator is only used here: > > > core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/InstanceIdAssigner.scala > > what is exactly the purpose of this class? > > From what I can undestand, it is storing in zooker the id of the invoker > for the purpose of guaranteeing > > Can we replace with something else that does not use zookeeper? We have > now etcd for the scheduler if I am not wrong. Can we use etcd and get rid > of the dependency on zookeepr 3.4 that holds back also the version of > kafka, so we can experiment with Kafka Kraft and RedPanda (that does not > have zookeeper at all)? > > -- > Michele Sciabarra > mich...@sciabarra.com >
