Re: R/W traffic estimation between Flink and Zookeeper

Great, thanks for the info, Stefan. On Thu, Nov 16, 2017, 01:59 Stefan Richter wrote: > Hi, > > I think Zookeeper is only used as a meta data store in HA mode. > Interactions with ZK are not part of the per-record stream processing code > paths of Flink. Things that

Re: R/W traffic estimation between Flink and Zookeeper

Hi, I think Zookeeper is only used as a meta data store in HA mode. Interactions with ZK are not part of the per-record stream processing code paths of Flink. Things that are written to ZK can (also depending on your job) include e.g. the job graph, Kafka offsets, or the meta data about

Re: R/W traffic estimation between Flink and Zookeeper

Thanks Piotr, does Flink read/write to zookeeper every time it process a record? I thought only JM uses ZK to keep some meta level data, not sure why `it depends on many things like state backend used, state size, complexity of your application, size of the records, number of machines, their

Re: R/W traffic estimation between Flink and Zookeeper

Hi, Are you asking how to measure records/s or is it possible to achieve it? To measure it you can check numRecordsInPerSecond metric. As far if 1000 records/s is possible, it depends on many things like state backend used, state size, complexity of your application, size of the records,

R/W traffic estimation between Flink and Zookeeper

Hi Is there a way to estimate read/write traffic between flink and zk? I am looking for something like 1000 reads/sec or 1000 writes/sec. And the size of the message. Thanks