Hi Adam No problem, I'm glad you had a good break :)
I think the goals and tradeoffs that Hermes chose provide a useful system, I'm just not sure reliable is quite the right word to use when describing it. When I think of 'reliable' in the context of a message queue I think of guaranteed delivery once the message is accepted (subject to some limitations like a quorum of nodes staying up). I get the other meaning of reliable that you're using - "delivered within a predictable time frame", but I wonder if something more like "predictable, bounded response times" would be more accurate (perhaps there's a catchier way of saying it :)) Documenting the tradeoffs and failure scenarios will make it easier for people to understand whether it fits their needs. I can imagine that you could couple Hermes with an offline reconciliation to check any 201'd messages were successfully delivered to Kafka. Anyway, great work on this and thanks for sharing! On Mon, 25 May 2015 at 1:26 am Adam Dubiel <dubiel.a...@gmail.com> wrote: > Hi Daniel, > > First of all sorry for late response, i enjoyed short vacation :) > > I guess the documentation might be bit misleading here, and so we should > improve it: we do not aim (and can't) provide higher guarantees than Kafka. > > We want to be as bullteproofs as possible in REST interface segments. In > our SLA we concentrate a lot on REST availability and response time. We can > also withstand some short-term Kafka outages. Still, the only goal of > Hermes frontend is to push event to Kafka, as only this way we can assure > our customers that the event will be delivered (and is stored reliably). > Thus, we do not plan on making any distributed storage here - Kafka is our > storage. > > Adam > > > > 2015-05-20 11:49 GMT+02:00 Daniel Compton <daniel.compton.li...@gmail.com > >: > > > Hi Adam > > > > Firstly, thanks for open sourcing this, it looks like a great tool and I > > can imagine a lot of people will find it very useful. > > > > I had a few thoughts reading the docs. I may have misunderstood things > but > > it seems that your goal of meeting a strict SLA conflicts with your goal > of > > bulletproof delivery. Even if you have a durable queue on disk, when you > > send a 202 Accepted you could still lose messages if you lost the disk > > storing the data. > > > > I realise this has a small chance of happening, but I don't think you can > > guarantee bulletproof delivery if only a single server stores the message > > while its in transit before being accepted by Kafka. > > > > Could you expand on the reliability guarantees you're looking to offer > and > > how they can be stronger than the ones provided by Kafka? > > > > Thanks, Daniel. > > On Tue, 19 May 2015 at 2:57 am Adam Dubiel <dubiel.a...@gmail.com> > wrote: > > > > > Hello everyone, > > > > > > I'm technical team lead of Hermes project. I will try to answer already > > > posted questions, but feel free to ask me anything. > > > > > > 1) Can you comment on how this compares to Confluent's REST proxy? > > > > > > We do not perceive Hermes as mere proxy. While Confluent product wants > to > > > help services written in non-jvm languages in connecting to Kafka, > Hermes > > > is more then that. First of all we wanted to make connecting to PubSub > as > > > easy as possible, hence REST API for publishing (this is same as REST > > > proxy), but also converting from pull to push on consumer side. This > > means, > > > that our consumers don't need to include Kafka drivers, handle retries > or > > > worry about linear commit offset. Instead, our hermes-consumer module > > > implements retry strategies and adaptive output-rate algorithm. > Consumer > > > needs to register HTTP (or JMS) endpoint that can accept POST message > and > > > that is all. > > > > > > Secondly, on the publisher side, we wanted to create somewhat > > bullet-proof > > > solution capable of accepting very sensitive data. This means we use > > Kafka > > > producer buffer to store unsent messages (our internal solution also > > > persists this buffer to disk, this will be ported to OpenSource soon) > and > > > we can guarantee maximum response time (SLA). We also are able to use > > > different levels of Kafka ACKs per topic (-1 or 1 currently). > > > > > > Last but not least, we mitigate some of administrative issues: added > tons > > > of metrics that can be reported to graphite, message state tracking for > > > debugging purposes, easy to use REST API for previewing messages stored > > at > > > Kafka or to retransmit events starting from given timestamp (not > > offset!), > > > > > > > > > 2) Performance > > > > > > We plan on making tests public, but they are not there yet. The numbers > > in > > > docs come from our production environment, but they should be taken > with > > > grain of salt. Hermes performance depends highly on underlying Kafka > > > cluster performance. Our cluster is deployed in cloud (on SSDs), bare > > metal > > > deployments would probably achieve a lot better performance. Still, the > > > most important metric here is not total response time, but Hermes > > overhead > > > over pure Kafka. Looks like this is neglible in our cloud deployment > > (p99 < > > > 0.2ms), but we will be crunching those numbers and publish them in > docs. > > > > > > 3) Topics/subscriptions limit > > > > > > We are limited by Kafka as well, though we never encountered any > problems > > > with this (still, we have only 100-150 topics). We want to scale out by > > > making Hermes multi-kafka aware (in effort to become multi DC). > Currently > > > management node can manage multiple Kafka clusters, but as soon as we > > > deploy it on production we will add some more documentation on > > architecture > > > and deployment. > > > > > > > > > We should create FAQ that would answer some most popular questions. > > > > > > > > > 2015-05-18 13:14 GMT+02:00 Marcin Kuthan <marcin.kut...@gmail.com>: > > > > > > > Hi Warren > > > > > > > > With Hermes, the publisher is never blocked. Even if message has not > > > > been sent to Kafka cluster, or if message has been sent but not > > > > acknowledged. It is especially useful if your system needs to have > > > > strict SLA guarantees. > > > > > > > > From the consumers perspective there is retrying policy if the > > > > consumer is not able to handle published message. In addition, Hermes > > > > is able to adjust speed of sending messages to actual situation (i.e. > > > > subscribing service has slown down due to heavy GC) > > > > > > > > > > > > On 18 May 2015 at 09:56, Warren Henning <warren.henn...@gmail.com> > > > wrote: > > > > > Interesting. Thanks for sharing and working on this! > > > > > > > > > > Can you comment on how this compares to Confluent's REST proxy? > > > > > > > > > > On Sat, May 16, 2015 at 12:01 AM, Marcin Kuthan < > > > marcin.kut...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > >> Hi Everyone > > > > >> > > > > >> Hermes is an asynchronous message broker built on top of Kafka. It > > > > >> provides reliable, fault tolerant REST interface for message > > > > >> publishing and adaptive push mechanisms for message sending. > Hermes > > is > > > > >> used as a message broker for asynchronous communication between > > > > >> microservices. > > > > >> > > > > >> Some of the main features: > > > > >> > > > > >> - Performance and scalability - Hermes in production handling up > to > > > > >> 30.000 msgs/sec with 99.9th percentile latency below 100 ms on a > > > > >> 4-node cluster. > > > > >> > > > > >> - Reliability - Hermes is used for publishing sensitive data, such > > as > > > > >> billing events, user account changes etc. Hermes allows to define > > more > > > > >> reliable policy for those important events - require acknowledge > > from > > > > >> all Kafka replicas and increase request timeouts. > > > > >> > > > > >> - Push model - It makes receiving messages from Hermes dead > simple: > > > > >> you just have to write one HTTP endpoint in your service. It’s up > to > > > > >> Hermes to create Kafka consumer, redeliver messages, keep eye on > > > > >> throughput limits etc. > > > > >> > > > > >> Feedback and comments are welcome, you can find Hermes > documentation > > > at: > > > > >> http://hermes-pubsub.readthedocs.org/en/latest/index.html > > > > >> > > > > >> Hermes is published under Apache Licence: > > > > >> https://github.com/allegro/hermes > > > > >> > > > > >> Best Regards, > > > > >> Marcin > > > > >> > > > > > > > > > >
