We use kafka for inter-topology communication. Going from input -> kafka -> storm -> kafka -> output takes around 20-40ms. Although you need to use kafka 0.8 or greater, as previous versions force an fsync on every message, which makes performance plummet.
The main reasons we went for kafka rather than DRPC was that kafka gives us a permanent log of all the messages in and out of the system, and that messages won't be lost on a server failure (whereas they might be with DRPC, as that just uses an in-memory queue) SimonC From: Jon Logan [mailto:jmlo...@buffalo.edu] Sent: 04 April 2014 02:44 To: user@storm.incubator.apache.org Subject: Replacing DRPC With Kafka Has anyone attempted to replace Storm DRPC with Kafka? My main concern stems from the weight of Kafka topics...especially for the handling of return results to clients.
