In regards to embedding an HTTP server in Kafka to handle REST requests, how hard would it be to modify/extend the existing SocketServer? Seems like lots of good work went into the networking stuff in Kafka, it would make sense to try and leverage that. I could imagine KafkaServer (optionally) starting up an HttpSocketServer with HttpRequestHandlers (similar to the existing SocketServer/RequestHandlers). Is this feasible/sensible?
Assuming Kafka handles the socket layer, perhaps something like Apache HTTP Components (http://hc.apache.org) could be used for parsing the HTTP messages? I'd like to stay away from higher level web frameworks for something simple like this. On Aug 6, 2012, at 2:19 PM, Jonathan Creasy wrote: > That is correct, a Consume REST request would be handed off to a Consumer > class which communicates with ZooKeeper and Kafka. The proxy keeps track of > clients, there are a few routes to manipulate the offsets if you wanted to > force a reset, seek to the end, or re-retrieve some data from a topic. > > If others agree, I would be happy to collaborate on integrating a REST > interface to Kakfa directly, more as your have proposed, rather than a proxy. > The proxy was low-hanging fruit within Box because we already had a framework > for cranking out a REST API quite easily and a set of Kafka+ZK classes. It > was simply stitching the two together. > > -Jonathan > > On Mon, Aug 6, 2012 at 5:39 AM, David Arthur <[email protected]> wrote: > I'd be happy to collaborate on this, though it's been a while since I've used > PHP. > > From what it looks like, what you have is a true proxy that runs outside of > Kafka and translates some REST routes into Kafka client calls. This sounds > more in line with what the project page describes. What I have proposed is > more like a translation layer between some REST routes and FetchRequests. In > this case the client is responsible for managing offsets. Using the consumer > groups and ZooKeeper would be another nice way of consuming messages (which > is probably more like what you have). > > Any maintainers have feedback on this? > > On Aug 3, 2012, at 4:13 PM, Jonathan Creasy wrote: > > > I have an internal one working and was hoping to have it open sourced in > > the next week. The one at Box is based on the CodeIgniter framework, we > > have about 45 RESTful interfaces built on this framework so I just put > > together another one for Kafka. > > > > > > Here are my notes, these were pre-dev so may be a little different than > > what we ended up with. > > > > https://cwiki.apache.org/confluence/display/KAFKA/Restful+API+Proposal > > > > I will read yours later this afternoon, we should work together. > > > > -Jonathan > > > > > > On Fri, Aug 3, 2012 at 7:41 AM, David Arthur <[email protected]> wrote: > > > >> I'd like to tackle this project (assuming it hasn't been started yet). > >> > >> I wrote up some initial thoughts here: https://gist.github.com/3248179 > >> > >> TLDR; use Range header for specifying offsets, simple URIs like > >> /kafka/topics/[topic]/[partition], use for a simple transport of bytes > >> and/or represent the messages as some media type (text, json, xml) > >> > >> Feedback is most welcome (in the Gist or in this thread). > >> > >> Cheers! > >> > >> -David > >
