Hemanth,
Can you be a bit more specific about your setup? Do you have control over
the format of the request bodies that reach HAProxy or not? If you do,
Confluent's REST proxy should work fine and does not require the Schema
Registry. It supports both binary (encoded as base64 so it can be passed
via the JSON request body) and Avro. With Avro it uses the schema registry,
but the binary mode doesn't require it.
If you don't have control over the format, then the REST proxy is not
currently designed to support that use case. I don't think HAProxy can
rewrite request bodies (beyond per-line regexes, which would be hard to
make work), so that's not an option either. It would certainly be possible
to make a small addition to the REST proxy to allow binary request bodies
to be produced directly to a topic specified in the URL, though you'd be
paying pretty high overhead per message -- without the ability to batch,
you're doing one HTTP request per messages. This might not be bad if your
messages are large enough? (Then again, the same issue applies regardless
of what solution you end up with if each of the requests to HAProxy only
contains one message).
-Ewen
On Wed, Aug 26, 2015 at 5:05 PM, Hemanth Abbina <heman...@eiqnetworks.com>
wrote:
> Marc,
>
> Thanks for your response. Let's have more details on the problem.
>
> As I already mentioned in the previous post, here is our expected data
> flow: logs -> HAProxy -> {new layer } -> Kafka Cluster
>
> The 'new layer' should receive logs as HTTP requests from HAproxy and
> produce the same logs to Kafka without loss.
>
> Options that seems to be available, are
> 1. Flume: It has a HTTP source & Kafka sink, but the documentation says
> HTTP source is not for production use.
> 2. Kafka Rest Proxy: Though this seems to be fine, adding another
> dependency of Schema Registry servers to validate the schema, which should
> be again used by the consumers.
> 3. Custom plugin to handle this functionality: Though the functionality
> seems to be simple - scalability, reliability aspects and maintenance would
> be more.
>
> Thanks
> Hemanth
>
> -----Original Message-----
> From: Marc Bollinger [mailto:m...@lumoslabs.com]
> Sent: Thursday, August 27, 2015 4:39 AM
> To: users@kafka.apache.org
> Cc: dev-subscr...@kafka.apache.org
> Subject: Re: Http Kafka producer
>
> I'm actually also really interested in this...I had a chat about this on
> the distributed systems slack's <http://dist-sys.slack.com> Kafka channel
> a few days ago, but we're not much further than griping about the problem.
> We're basically migrating an existing event system, one which packed
> messages into files, waited for a time-or-space threshold to be crossed,
> then dealt with distribution in terms of files. Basically, we'd like to
> keep a lot of those semantics: we can acknowledge success on the app server
> as soon as we've flushed to disk, and rely on the filesystem for
> durability, and total order across the system doesn't matter, as the HTTP
> PUTs sending the messages are load balanced across many app servers. We
> also can tolerate [very] long downstream event system outages,
> because...we're ultimately just writing sequentially to disk, per process
> (I should mention that this part is in Rails, which means we're dealing
> largely in terms of processes, not threads).
>
> RocksDB was mentioned in the discussion, but spending exactly 5 minutes
> researching that solution, it seems like the dead simplest solution on an
> app server in terms of moving parts (multiple processes writing, one
> process reading/forwarding to Kafka) wouldn't work well with RocksDB.
> Although now that I'm looking at it more, it looks like they're working on
> a MySQL storage engine?
>
> Anyway yeah, I'd love some discussion on this, or war stories of migration
> to Kafka from other event systems (F/OSS or...bespoke).
>
> On Wed, Aug 26, 2015 at 3:45 PM, Hemanth Abbina <heman...@eiqnetworks.com>
> wrote:
>
> > Hi,
> >
> > Our application receives events through a HAProxy server on HTTPs,
> > which should be forwarded and stored to Kafka cluster.
> >
> > What should be the best option for this ?
> > This layer should receive events from HAProxy & produce them to Kafka
> > cluster, in a reliable and efficient way (and should scale horizontally).
> >
> > Please suggest.
> >
> > --regards
> > Hemanth
> >
>
--
Thanks,
Ewen
