Hi Siyuan, 1) new consumer is single-threaded, it does not maintain any internal threads as the old high-level consumer.
2) each consumer will only maintain one TCP connection with each broker. The only extra socket is the one with its coordinator. That is, if there is three brokers S1, S2, S3, and S1 is the coordinator for this consumer, it will maintain 4 sockets in total, 2 for S1 (one for fetching, one for coordinating) and 1 for S2 and S3 (only for fetching). 3) Currently the connection is not closed by consumer, although the underlying network client / selector will close idle ones after some timeout. So in worst case it will only maintain N+1 sockets in total for N Kafka brokers at one time. Guozhang On Mon, Nov 16, 2015 at 4:22 PM, hsy...@gmail.com <hsy...@gmail.com> wrote: > The new consumer API looks good. If I understand it correctly you can use > it like simple consumer or high-level consumer. But I have couple questions > about it's internal implementation > > First of all does the consumer have any internal fetcher threads like > high-level consumer? > > When you assign multiple TopicPartitions to a consumer, how many TCP > connections it establish to the brokers. Is it same as number of leader > brokers that host those partitions or just number of TopicPartitions. If > there is any leader broker change does it establish new connections/using > existing connections to fetch the data? Can it continue consuming? Also is > the connection kept until the consumer is closed? > > Thanks! > > Best, > Siyuan > -- -- Guozhang
