[
https://issues.apache.org/jira/browse/KAFKA-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14365766#comment-14365766
]
Jay Kreps commented on KAFKA-1912:
----------------------------------
I think this is easier than I thought.
We have an async api now in NetworkClient:
send(ClientRequest request)
So re-routing is fairly straight-forward:
reroutedRequest = new RequestSend(newDestinationNode, originalRequest.header,
originalRequest.body);
networkClient.send(new ClientRequest(time, true, reroutedRequest,
new RequestCompletionHandler() {
public void onComplete(ClientResponse response) {
responseQueue.add(response);
}
}
));
The only issue with this is that the network client currently requires a ready
connection to do a send. I think we can fix this by making a small
RequestRerouter shim that has a queue of unsent messages and a thread that does
the ready check and send. Hopefully that makes sense, if not I can flesh it out
a bit more.
> Create a simple request re-routing facility
> -------------------------------------------
>
> Key: KAFKA-1912
> URL: https://issues.apache.org/jira/browse/KAFKA-1912
> Project: Kafka
> Issue Type: Improvement
> Reporter: Jay Kreps
> Fix For: 0.8.3
>
>
> We are accumulating a lot of requests that have to be directed to the correct
> server. This makes sense for high volume produce or fetch requests. But it is
> silly to put the extra burden on the client for the many miscellaneous
> requests such as fetching or committing offsets and so on.
> This adds a ton of practical complexity to the clients with little or no
> payoff in performance.
> We should add a generic request-type agnostic re-routing facility on the
> server. This would allow any server to accept a request and forward it to the
> correct destination, proxying the response back to the user. Naturally it
> needs to do this without blocking the thread.
> The result is that a client implementation can choose to be optimally
> efficient and manage a local cache of cluster state and attempt to always
> direct its requests to the proper server OR it can choose simplicity and just
> send things all to a single host and let that host figure out where to
> forward it.
> I actually think we should implement this more or less across the board, but
> some requests such as produce and fetch require more logic to proxy since
> they have to be scattered out to multiple servers and gathered back to create
> the response. So these could be done in a second phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
