Found out that there is standard API for retrieving and committing offsets
(see
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
)
Problem is that the server/broker side is not extensible (see
Hi,
I haven't followed the changes to offset tracking closely, other than that
storing them in ZK is not the only option any more.
I think what Stevo is asking about/suggesting is that there there be a
single API from which offset information can be retrieved (e.g. by
monitoring tools), so that
Stevo,
Both of the main solutions used by the high-level consumer are standardized
and supported directly by the kafka client libraries (e.g. maintaining
offsets in zookeeper or in kafka itself). And for the zk case, there is
the consumer offset checker (which is good for monitoring). Consumer
I guess adding a new component will increase the complexity of the system
structure. And if the new component consists of one or a few nodes, it may
becomes the bottleneck of the whole system, if it consists of many nodes,
it will make the system even more complex.
Although every solution has its
It could be a separate server component, does not have to be
monolith/coupled with broker.
Such solution would have benefits - single API, pluggable implementations.
On Wed, May 27, 2015 at 8:57 AM, Shady Xu shad...@gmail.com wrote:
Storing and managing offsets by broker will leave high
Hello Apache Kafka community,
Please correct me if wrong, AFAIK currently (Kafka 0.8.2.x) offset
management responsibility is mainly client/consumer side responsibility.
Wouldn't it be better if it was broker side only responsibility?
E.g. now if one wants to use custom offset management, any
