Hi, Jiangjie, Thanks for taking on this.
I was thinking that one way to decouple the dependency on Metadata in NetworkClient is the following. 1. Make Metadata an interface. 2. Rename current Metadata class to sth like KafkaMetadata that implements the Metadata interface. 3. Have a new NoOpMetadata class that implements the Metadata interface. This class 3.1 does nothing for any write method 3.2 returns max long for any method that asks for a timestamp 3.3. returns an empty Cluster for fetch(). Then we can leave NetworkClient unchanged and just pass in a NoOpMetadata when using NetworkClient in the controller. The consumer/producer client will be using KafkaMetadata. As for replica fetchers, it may be possible to use KafkaConsumer. However, we don't need the metadata and the offset management. So, perhaps it's easier to just use NetworkClient. Also, currently, there is one replica fetcher thread per source broker. By using NetworkClient, we can change that to using a single thread for all source brokers. This is probably a bigger change. So, maybe we can do it later. Jun I think we probably need to replace replica fetcher with NetworkClient as well. Replica fetcher gets leader from the controller and therefore doesn't On Tue, May 5, 2015 at 1:37 PM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > I am trying to see if we can reuse the NetworkClient class to be used in > controller to broker communication. (Also, we can probably use > KafkaConsumer which is already using NetworkClient in replica fetchers). > Currently NetworkClient does the following things in addition to sending > requests. > > 1. Connection state management. > 2. Flow control (inflight requests) > 3. Metadata refresh > > In controller we need (1) and (2) but not (3). NetworkClient is tightly > coupled with metadata now and this is the major blocker of reusing > NetworkClient in controller. For controller, we don’t need NetworkClient > to manage any metadata because the controller has listeners to monitor the > cluster state and has all the information about topic metadata. > I am thinking we can add a disable metadata refresh flag to NetworkClient > or set metadata refresh interval to be Long.MAX_VALUE, so the metadata will > be managed outside NetworkClient. > This needs minimal change to allow NetworkClient to be reused, but the > ugly part is NetworkClient still has the entire Metadata while it actually > only needs a NodeList. > > Want to see what do people think about this. > > Thanks. > > Jiangjie (Becket) Qin > >
