Re: [Announcement] Hermes - pub / sub broker built on top of Kafka
Thanks for the feedback. We will improve our documentation to make it clear what guarantees we concentrate on and what are the tradeoffs. And think of some catchy way to describe the system :) 2015-05-25 10:02 GMT+02:00 Daniel Compton : > Hi Adam > > No problem, I'm glad you had a good break :) > > I think the goals and tradeoffs that Hermes chose provide a useful system, > I'm just not sure reliable is quite the right word to use when describing > it. When I think of 'reliable' in the context of a message queue I think of > guaranteed delivery once the message is accepted (subject to some > limitations like a quorum of nodes staying up). > > I get the other meaning of reliable that you're using - "delivered within a > predictable time frame", but I wonder if something more like "predictable, > bounded response times" would be more accurate (perhaps there's a catchier > way of saying it :)) > > Documenting the tradeoffs and failure scenarios will make it easier for > people to understand whether it fits their needs. I can imagine that you > could couple Hermes with an offline reconciliation to check any 201'd > messages were successfully delivered to Kafka. > > Anyway, great work on this and thanks for sharing! > On Mon, 25 May 2015 at 1:26 am Adam Dubiel wrote: > > > Hi Daniel, > > > > First of all sorry for late response, i enjoyed short vacation :) > > > > I guess the documentation might be bit misleading here, and so we should > > improve it: we do not aim (and can't) provide higher guarantees than > Kafka. > > > > We want to be as bullteproofs as possible in REST interface segments. In > > our SLA we concentrate a lot on REST availability and response time. We > can > > also withstand some short-term Kafka outages. Still, the only goal of > > Hermes frontend is to push event to Kafka, as only this way we can assure > > our customers that the event will be delivered (and is stored reliably). > > Thus, we do not plan on making any distributed storage here - Kafka is > our > > storage. > > > > Adam > > > > > > > > 2015-05-20 11:49 GMT+02:00 Daniel Compton < > daniel.compton.li...@gmail.com > > >: > > > > > Hi Adam > > > > > > Firstly, thanks for open sourcing this, it looks like a great tool and > I > > > can imagine a lot of people will find it very useful. > > > > > > I had a few thoughts reading the docs. I may have misunderstood things > > but > > > it seems that your goal of meeting a strict SLA conflicts with your > goal > > of > > > bulletproof delivery. Even if you have a durable queue on disk, when > you > > > send a 202 Accepted you could still lose messages if you lost the disk > > > storing the data. > > > > > > I realise this has a small chance of happening, but I don't think you > can > > > guarantee bulletproof delivery if only a single server stores the > message > > > while its in transit before being accepted by Kafka. > > > > > > Could you expand on the reliability guarantees you're looking to offer > > and > > > how they can be stronger than the ones provided by Kafka? > > > > > > Thanks, Daniel. > > > On Tue, 19 May 2015 at 2:57 am Adam Dubiel > > wrote: > > > > > > > Hello everyone, > > > > > > > > I'm technical team lead of Hermes project. I will try to answer > already > > > > posted questions, but feel free to ask me anything. > > > > > > > > 1) Can you comment on how this compares to Confluent's REST proxy? > > > > > > > > We do not perceive Hermes as mere proxy. While Confluent product > wants > > to > > > > help services written in non-jvm languages in connecting to Kafka, > > Hermes > > > > is more then that. First of all we wanted to make connecting to > PubSub > > as > > > > easy as possible, hence REST API for publishing (this is same as REST > > > > proxy), but also converting from pull to push on consumer side. This > > > means, > > > > that our consumers don't need to include Kafka drivers, handle > retries > > or > > > > worry about linear commit offset. Instead, our hermes-consumer module > > > > implements retry strategies and adaptive output-rate algorithm. > > Consumer > > > > needs to register HTTP (or JMS) endpoint that can accept POST message > > and > > > > that is all. > > > > > > > > Secondly, on the publisher side, we wanted to create somewhat > > > bullet-proof > > > > solution capable of accepting very sensitive data. This means we use > > > Kafka > > > > producer buffer to store unsent messages (our internal solution also > > > > persists this buffer to disk, this will be ported to OpenSource soon) > > and > > > > we can guarantee maximum response time (SLA). We also are able to use > > > > different levels of Kafka ACKs per topic (-1 or 1 currently). > > > > > > > > Last but not least, we mitigate some of administrative issues: added > > tons > > > > of metrics that can be reported to graphite, message state tracking > for > > > > debugging purposes, easy to use REST API for previewing messages > stored > > > at > > > > Kafka or to retransmit events
Re: [Announcement] Hermes - pub / sub broker built on top of Kafka
Hi Adam No problem, I'm glad you had a good break :) I think the goals and tradeoffs that Hermes chose provide a useful system, I'm just not sure reliable is quite the right word to use when describing it. When I think of 'reliable' in the context of a message queue I think of guaranteed delivery once the message is accepted (subject to some limitations like a quorum of nodes staying up). I get the other meaning of reliable that you're using - "delivered within a predictable time frame", but I wonder if something more like "predictable, bounded response times" would be more accurate (perhaps there's a catchier way of saying it :)) Documenting the tradeoffs and failure scenarios will make it easier for people to understand whether it fits their needs. I can imagine that you could couple Hermes with an offline reconciliation to check any 201'd messages were successfully delivered to Kafka. Anyway, great work on this and thanks for sharing! On Mon, 25 May 2015 at 1:26 am Adam Dubiel wrote: > Hi Daniel, > > First of all sorry for late response, i enjoyed short vacation :) > > I guess the documentation might be bit misleading here, and so we should > improve it: we do not aim (and can't) provide higher guarantees than Kafka. > > We want to be as bullteproofs as possible in REST interface segments. In > our SLA we concentrate a lot on REST availability and response time. We can > also withstand some short-term Kafka outages. Still, the only goal of > Hermes frontend is to push event to Kafka, as only this way we can assure > our customers that the event will be delivered (and is stored reliably). > Thus, we do not plan on making any distributed storage here - Kafka is our > storage. > > Adam > > > > 2015-05-20 11:49 GMT+02:00 Daniel Compton >: > > > Hi Adam > > > > Firstly, thanks for open sourcing this, it looks like a great tool and I > > can imagine a lot of people will find it very useful. > > > > I had a few thoughts reading the docs. I may have misunderstood things > but > > it seems that your goal of meeting a strict SLA conflicts with your goal > of > > bulletproof delivery. Even if you have a durable queue on disk, when you > > send a 202 Accepted you could still lose messages if you lost the disk > > storing the data. > > > > I realise this has a small chance of happening, but I don't think you can > > guarantee bulletproof delivery if only a single server stores the message > > while its in transit before being accepted by Kafka. > > > > Could you expand on the reliability guarantees you're looking to offer > and > > how they can be stronger than the ones provided by Kafka? > > > > Thanks, Daniel. > > On Tue, 19 May 2015 at 2:57 am Adam Dubiel > wrote: > > > > > Hello everyone, > > > > > > I'm technical team lead of Hermes project. I will try to answer already > > > posted questions, but feel free to ask me anything. > > > > > > 1) Can you comment on how this compares to Confluent's REST proxy? > > > > > > We do not perceive Hermes as mere proxy. While Confluent product wants > to > > > help services written in non-jvm languages in connecting to Kafka, > Hermes > > > is more then that. First of all we wanted to make connecting to PubSub > as > > > easy as possible, hence REST API for publishing (this is same as REST > > > proxy), but also converting from pull to push on consumer side. This > > means, > > > that our consumers don't need to include Kafka drivers, handle retries > or > > > worry about linear commit offset. Instead, our hermes-consumer module > > > implements retry strategies and adaptive output-rate algorithm. > Consumer > > > needs to register HTTP (or JMS) endpoint that can accept POST message > and > > > that is all. > > > > > > Secondly, on the publisher side, we wanted to create somewhat > > bullet-proof > > > solution capable of accepting very sensitive data. This means we use > > Kafka > > > producer buffer to store unsent messages (our internal solution also > > > persists this buffer to disk, this will be ported to OpenSource soon) > and > > > we can guarantee maximum response time (SLA). We also are able to use > > > different levels of Kafka ACKs per topic (-1 or 1 currently). > > > > > > Last but not least, we mitigate some of administrative issues: added > tons > > > of metrics that can be reported to graphite, message state tracking for > > > debugging purposes, easy to use REST API for previewing messages stored > > at > > > Kafka or to retransmit events starting from given timestamp (not > > offset!), > > > > > > > > > 2) Performance > > > > > > We plan on making tests public, but they are not there yet. The numbers > > in > > > docs come from our production environment, but they should be taken > with > > > grain of salt. Hermes performance depends highly on underlying Kafka > > > cluster performance. Our cluster is deployed in cloud (on SSDs), bare > > metal > > > deployments would probably achieve a lot better performance. Still, the > > > most important metri
Re: [Announcement] Hermes - pub / sub broker built on top of Kafka
Hi Daniel, First of all sorry for late response, i enjoyed short vacation :) I guess the documentation might be bit misleading here, and so we should improve it: we do not aim (and can't) provide higher guarantees than Kafka. We want to be as bullteproofs as possible in REST interface segments. In our SLA we concentrate a lot on REST availability and response time. We can also withstand some short-term Kafka outages. Still, the only goal of Hermes frontend is to push event to Kafka, as only this way we can assure our customers that the event will be delivered (and is stored reliably). Thus, we do not plan on making any distributed storage here - Kafka is our storage. Adam 2015-05-20 11:49 GMT+02:00 Daniel Compton : > Hi Adam > > Firstly, thanks for open sourcing this, it looks like a great tool and I > can imagine a lot of people will find it very useful. > > I had a few thoughts reading the docs. I may have misunderstood things but > it seems that your goal of meeting a strict SLA conflicts with your goal of > bulletproof delivery. Even if you have a durable queue on disk, when you > send a 202 Accepted you could still lose messages if you lost the disk > storing the data. > > I realise this has a small chance of happening, but I don't think you can > guarantee bulletproof delivery if only a single server stores the message > while its in transit before being accepted by Kafka. > > Could you expand on the reliability guarantees you're looking to offer and > how they can be stronger than the ones provided by Kafka? > > Thanks, Daniel. > On Tue, 19 May 2015 at 2:57 am Adam Dubiel wrote: > > > Hello everyone, > > > > I'm technical team lead of Hermes project. I will try to answer already > > posted questions, but feel free to ask me anything. > > > > 1) Can you comment on how this compares to Confluent's REST proxy? > > > > We do not perceive Hermes as mere proxy. While Confluent product wants to > > help services written in non-jvm languages in connecting to Kafka, Hermes > > is more then that. First of all we wanted to make connecting to PubSub as > > easy as possible, hence REST API for publishing (this is same as REST > > proxy), but also converting from pull to push on consumer side. This > means, > > that our consumers don't need to include Kafka drivers, handle retries or > > worry about linear commit offset. Instead, our hermes-consumer module > > implements retry strategies and adaptive output-rate algorithm. Consumer > > needs to register HTTP (or JMS) endpoint that can accept POST message and > > that is all. > > > > Secondly, on the publisher side, we wanted to create somewhat > bullet-proof > > solution capable of accepting very sensitive data. This means we use > Kafka > > producer buffer to store unsent messages (our internal solution also > > persists this buffer to disk, this will be ported to OpenSource soon) and > > we can guarantee maximum response time (SLA). We also are able to use > > different levels of Kafka ACKs per topic (-1 or 1 currently). > > > > Last but not least, we mitigate some of administrative issues: added tons > > of metrics that can be reported to graphite, message state tracking for > > debugging purposes, easy to use REST API for previewing messages stored > at > > Kafka or to retransmit events starting from given timestamp (not > offset!), > > > > > > 2) Performance > > > > We plan on making tests public, but they are not there yet. The numbers > in > > docs come from our production environment, but they should be taken with > > grain of salt. Hermes performance depends highly on underlying Kafka > > cluster performance. Our cluster is deployed in cloud (on SSDs), bare > metal > > deployments would probably achieve a lot better performance. Still, the > > most important metric here is not total response time, but Hermes > overhead > > over pure Kafka. Looks like this is neglible in our cloud deployment > (p99 < > > 0.2ms), but we will be crunching those numbers and publish them in docs. > > > > 3) Topics/subscriptions limit > > > > We are limited by Kafka as well, though we never encountered any problems > > with this (still, we have only 100-150 topics). We want to scale out by > > making Hermes multi-kafka aware (in effort to become multi DC). Currently > > management node can manage multiple Kafka clusters, but as soon as we > > deploy it on production we will add some more documentation on > architecture > > and deployment. > > > > > > We should create FAQ that would answer some most popular questions. > > > > > > 2015-05-18 13:14 GMT+02:00 Marcin Kuthan : > > > > > Hi Warren > > > > > > With Hermes, the publisher is never blocked. Even if message has not > > > been sent to Kafka cluster, or if message has been sent but not > > > acknowledged. It is especially useful if your system needs to have > > > strict SLA guarantees. > > > > > > From the consumers perspective there is retrying policy if the > > > consumer is not able to handle published message.
Re: [Announcement] Hermes - pub / sub broker built on top of Kafka
Hi Adam Firstly, thanks for open sourcing this, it looks like a great tool and I can imagine a lot of people will find it very useful. I had a few thoughts reading the docs. I may have misunderstood things but it seems that your goal of meeting a strict SLA conflicts with your goal of bulletproof delivery. Even if you have a durable queue on disk, when you send a 202 Accepted you could still lose messages if you lost the disk storing the data. I realise this has a small chance of happening, but I don't think you can guarantee bulletproof delivery if only a single server stores the message while its in transit before being accepted by Kafka. Could you expand on the reliability guarantees you're looking to offer and how they can be stronger than the ones provided by Kafka? Thanks, Daniel. On Tue, 19 May 2015 at 2:57 am Adam Dubiel wrote: > Hello everyone, > > I'm technical team lead of Hermes project. I will try to answer already > posted questions, but feel free to ask me anything. > > 1) Can you comment on how this compares to Confluent's REST proxy? > > We do not perceive Hermes as mere proxy. While Confluent product wants to > help services written in non-jvm languages in connecting to Kafka, Hermes > is more then that. First of all we wanted to make connecting to PubSub as > easy as possible, hence REST API for publishing (this is same as REST > proxy), but also converting from pull to push on consumer side. This means, > that our consumers don't need to include Kafka drivers, handle retries or > worry about linear commit offset. Instead, our hermes-consumer module > implements retry strategies and adaptive output-rate algorithm. Consumer > needs to register HTTP (or JMS) endpoint that can accept POST message and > that is all. > > Secondly, on the publisher side, we wanted to create somewhat bullet-proof > solution capable of accepting very sensitive data. This means we use Kafka > producer buffer to store unsent messages (our internal solution also > persists this buffer to disk, this will be ported to OpenSource soon) and > we can guarantee maximum response time (SLA). We also are able to use > different levels of Kafka ACKs per topic (-1 or 1 currently). > > Last but not least, we mitigate some of administrative issues: added tons > of metrics that can be reported to graphite, message state tracking for > debugging purposes, easy to use REST API for previewing messages stored at > Kafka or to retransmit events starting from given timestamp (not offset!), > > > 2) Performance > > We plan on making tests public, but they are not there yet. The numbers in > docs come from our production environment, but they should be taken with > grain of salt. Hermes performance depends highly on underlying Kafka > cluster performance. Our cluster is deployed in cloud (on SSDs), bare metal > deployments would probably achieve a lot better performance. Still, the > most important metric here is not total response time, but Hermes overhead > over pure Kafka. Looks like this is neglible in our cloud deployment (p99 < > 0.2ms), but we will be crunching those numbers and publish them in docs. > > 3) Topics/subscriptions limit > > We are limited by Kafka as well, though we never encountered any problems > with this (still, we have only 100-150 topics). We want to scale out by > making Hermes multi-kafka aware (in effort to become multi DC). Currently > management node can manage multiple Kafka clusters, but as soon as we > deploy it on production we will add some more documentation on architecture > and deployment. > > > We should create FAQ that would answer some most popular questions. > > > 2015-05-18 13:14 GMT+02:00 Marcin Kuthan : > > > Hi Warren > > > > With Hermes, the publisher is never blocked. Even if message has not > > been sent to Kafka cluster, or if message has been sent but not > > acknowledged. It is especially useful if your system needs to have > > strict SLA guarantees. > > > > From the consumers perspective there is retrying policy if the > > consumer is not able to handle published message. In addition, Hermes > > is able to adjust speed of sending messages to actual situation (i.e. > > subscribing service has slown down due to heavy GC) > > > > > > On 18 May 2015 at 09:56, Warren Henning > wrote: > > > Interesting. Thanks for sharing and working on this! > > > > > > Can you comment on how this compares to Confluent's REST proxy? > > > > > > On Sat, May 16, 2015 at 12:01 AM, Marcin Kuthan < > marcin.kut...@gmail.com > > > > > > wrote: > > > > > >> Hi Everyone > > >> > > >> Hermes is an asynchronous message broker built on top of Kafka. It > > >> provides reliable, fault tolerant REST interface for message > > >> publishing and adaptive push mechanisms for message sending. Hermes is > > >> used as a message broker for asynchronous communication between > > >> microservices. > > >> > > >> Some of the main features: > > >> > > >> - Performance and scalability - Hermes in production handling up to
Re: [Announcement] Hermes - pub / sub broker built on top of Kafka
Hello everyone, I'm technical team lead of Hermes project. I will try to answer already posted questions, but feel free to ask me anything. 1) Can you comment on how this compares to Confluent's REST proxy? We do not perceive Hermes as mere proxy. While Confluent product wants to help services written in non-jvm languages in connecting to Kafka, Hermes is more then that. First of all we wanted to make connecting to PubSub as easy as possible, hence REST API for publishing (this is same as REST proxy), but also converting from pull to push on consumer side. This means, that our consumers don't need to include Kafka drivers, handle retries or worry about linear commit offset. Instead, our hermes-consumer module implements retry strategies and adaptive output-rate algorithm. Consumer needs to register HTTP (or JMS) endpoint that can accept POST message and that is all. Secondly, on the publisher side, we wanted to create somewhat bullet-proof solution capable of accepting very sensitive data. This means we use Kafka producer buffer to store unsent messages (our internal solution also persists this buffer to disk, this will be ported to OpenSource soon) and we can guarantee maximum response time (SLA). We also are able to use different levels of Kafka ACKs per topic (-1 or 1 currently). Last but not least, we mitigate some of administrative issues: added tons of metrics that can be reported to graphite, message state tracking for debugging purposes, easy to use REST API for previewing messages stored at Kafka or to retransmit events starting from given timestamp (not offset!), 2) Performance We plan on making tests public, but they are not there yet. The numbers in docs come from our production environment, but they should be taken with grain of salt. Hermes performance depends highly on underlying Kafka cluster performance. Our cluster is deployed in cloud (on SSDs), bare metal deployments would probably achieve a lot better performance. Still, the most important metric here is not total response time, but Hermes overhead over pure Kafka. Looks like this is neglible in our cloud deployment (p99 < 0.2ms), but we will be crunching those numbers and publish them in docs. 3) Topics/subscriptions limit We are limited by Kafka as well, though we never encountered any problems with this (still, we have only 100-150 topics). We want to scale out by making Hermes multi-kafka aware (in effort to become multi DC). Currently management node can manage multiple Kafka clusters, but as soon as we deploy it on production we will add some more documentation on architecture and deployment. We should create FAQ that would answer some most popular questions. 2015-05-18 13:14 GMT+02:00 Marcin Kuthan : > Hi Warren > > With Hermes, the publisher is never blocked. Even if message has not > been sent to Kafka cluster, or if message has been sent but not > acknowledged. It is especially useful if your system needs to have > strict SLA guarantees. > > From the consumers perspective there is retrying policy if the > consumer is not able to handle published message. In addition, Hermes > is able to adjust speed of sending messages to actual situation (i.e. > subscribing service has slown down due to heavy GC) > > > On 18 May 2015 at 09:56, Warren Henning wrote: > > Interesting. Thanks for sharing and working on this! > > > > Can you comment on how this compares to Confluent's REST proxy? > > > > On Sat, May 16, 2015 at 12:01 AM, Marcin Kuthan > > > wrote: > > > >> Hi Everyone > >> > >> Hermes is an asynchronous message broker built on top of Kafka. It > >> provides reliable, fault tolerant REST interface for message > >> publishing and adaptive push mechanisms for message sending. Hermes is > >> used as a message broker for asynchronous communication between > >> microservices. > >> > >> Some of the main features: > >> > >> - Performance and scalability - Hermes in production handling up to > >> 30.000 msgs/sec with 99.9th percentile latency below 100 ms on a > >> 4-node cluster. > >> > >> - Reliability - Hermes is used for publishing sensitive data, such as > >> billing events, user account changes etc. Hermes allows to define more > >> reliable policy for those important events - require acknowledge from > >> all Kafka replicas and increase request timeouts. > >> > >> - Push model - It makes receiving messages from Hermes dead simple: > >> you just have to write one HTTP endpoint in your service. It’s up to > >> Hermes to create Kafka consumer, redeliver messages, keep eye on > >> throughput limits etc. > >> > >> Feedback and comments are welcome, you can find Hermes documentation at: > >> http://hermes-pubsub.readthedocs.org/en/latest/index.html > >> > >> Hermes is published under Apache Licence: > >> https://github.com/allegro/hermes > >> > >> Best Regards, > >> Marcin > >> >
Re: [Announcement] Hermes - pub / sub broker built on top of Kafka
Hi Warren With Hermes, the publisher is never blocked. Even if message has not been sent to Kafka cluster, or if message has been sent but not acknowledged. It is especially useful if your system needs to have strict SLA guarantees. >From the consumers perspective there is retrying policy if the consumer is not able to handle published message. In addition, Hermes is able to adjust speed of sending messages to actual situation (i.e. subscribing service has slown down due to heavy GC) On 18 May 2015 at 09:56, Warren Henning wrote: > Interesting. Thanks for sharing and working on this! > > Can you comment on how this compares to Confluent's REST proxy? > > On Sat, May 16, 2015 at 12:01 AM, Marcin Kuthan > wrote: > >> Hi Everyone >> >> Hermes is an asynchronous message broker built on top of Kafka. It >> provides reliable, fault tolerant REST interface for message >> publishing and adaptive push mechanisms for message sending. Hermes is >> used as a message broker for asynchronous communication between >> microservices. >> >> Some of the main features: >> >> - Performance and scalability - Hermes in production handling up to >> 30.000 msgs/sec with 99.9th percentile latency below 100 ms on a >> 4-node cluster. >> >> - Reliability - Hermes is used for publishing sensitive data, such as >> billing events, user account changes etc. Hermes allows to define more >> reliable policy for those important events - require acknowledge from >> all Kafka replicas and increase request timeouts. >> >> - Push model - It makes receiving messages from Hermes dead simple: >> you just have to write one HTTP endpoint in your service. It’s up to >> Hermes to create Kafka consumer, redeliver messages, keep eye on >> throughput limits etc. >> >> Feedback and comments are welcome, you can find Hermes documentation at: >> http://hermes-pubsub.readthedocs.org/en/latest/index.html >> >> Hermes is published under Apache Licence: >> https://github.com/allegro/hermes >> >> Best Regards, >> Marcin >>
Re: [Announcement] Hermes - pub / sub broker built on top of Kafka
Hi Stevo Thank's for your comment. On 16 May 2015 at 17:43, Stevo Slavić wrote: > Nice, thanks for sharing! > > Is 30k msgs/sec publishing or push throughput? Will check, hopefully > performance tests are included in sources. 30k msgs/sec is our regular traffic handled by hermes on 4-nodes cluster. 30k/sec messages published to the broker. > > Does Hermes have same max number of topics limitations as Kafka or does it > include a solution to have that aspect scalable as well? I'm interested how many topics/partitions are problematic for Kafka cluster? We did not encounter Kafka limitations for max number of topics in our pub/sub scenarios. > On May 16, 2015 8:02 AM, "Marcin Kuthan" wrote: > >> Hi Everyone >> >> Hermes is an asynchronous message broker built on top of Kafka. It >> provides reliable, fault tolerant REST interface for message >> publishing and adaptive push mechanisms for message sending. Hermes is >> used as a message broker for asynchronous communication between >> microservices. >> >> Some of the main features: >> >> - Performance and scalability - Hermes in production handling up to >> 30.000 msgs/sec with 99.9th percentile latency below 100 ms on a >> 4-node cluster. >> >> - Reliability - Hermes is used for publishing sensitive data, such as >> billing events, user account changes etc. Hermes allows to define more >> reliable policy for those important events - require acknowledge from >> all Kafka replicas and increase request timeouts. >> >> - Push model - It makes receiving messages from Hermes dead simple: >> you just have to write one HTTP endpoint in your service. It’s up to >> Hermes to create Kafka consumer, redeliver messages, keep eye on >> throughput limits etc. >> >> Feedback and comments are welcome, you can find Hermes documentation at: >> http://hermes-pubsub.readthedocs.org/en/latest/index.html >> >> Hermes is published under Apache Licence: >> https://github.com/allegro/hermes >> >> Best Regards, >> Marcin >>
Re: [Announcement] Hermes - pub / sub broker built on top of Kafka
Interesting. Thanks for sharing and working on this! Can you comment on how this compares to Confluent's REST proxy? On Sat, May 16, 2015 at 12:01 AM, Marcin Kuthan wrote: > Hi Everyone > > Hermes is an asynchronous message broker built on top of Kafka. It > provides reliable, fault tolerant REST interface for message > publishing and adaptive push mechanisms for message sending. Hermes is > used as a message broker for asynchronous communication between > microservices. > > Some of the main features: > > - Performance and scalability - Hermes in production handling up to > 30.000 msgs/sec with 99.9th percentile latency below 100 ms on a > 4-node cluster. > > - Reliability - Hermes is used for publishing sensitive data, such as > billing events, user account changes etc. Hermes allows to define more > reliable policy for those important events - require acknowledge from > all Kafka replicas and increase request timeouts. > > - Push model - It makes receiving messages from Hermes dead simple: > you just have to write one HTTP endpoint in your service. It’s up to > Hermes to create Kafka consumer, redeliver messages, keep eye on > throughput limits etc. > > Feedback and comments are welcome, you can find Hermes documentation at: > http://hermes-pubsub.readthedocs.org/en/latest/index.html > > Hermes is published under Apache Licence: > https://github.com/allegro/hermes > > Best Regards, > Marcin >
Re: [Announcement] Hermes - pub / sub broker built on top of Kafka
Nice, thanks for sharing! Is 30k msgs/sec publishing or push throughput? Will check, hopefully performance tests are included in sources. Does Hermes have same max number of topics limitations as Kafka or does it include a solution to have that aspect scalable as well? On May 16, 2015 8:02 AM, "Marcin Kuthan" wrote: > Hi Everyone > > Hermes is an asynchronous message broker built on top of Kafka. It > provides reliable, fault tolerant REST interface for message > publishing and adaptive push mechanisms for message sending. Hermes is > used as a message broker for asynchronous communication between > microservices. > > Some of the main features: > > - Performance and scalability - Hermes in production handling up to > 30.000 msgs/sec with 99.9th percentile latency below 100 ms on a > 4-node cluster. > > - Reliability - Hermes is used for publishing sensitive data, such as > billing events, user account changes etc. Hermes allows to define more > reliable policy for those important events - require acknowledge from > all Kafka replicas and increase request timeouts. > > - Push model - It makes receiving messages from Hermes dead simple: > you just have to write one HTTP endpoint in your service. It’s up to > Hermes to create Kafka consumer, redeliver messages, keep eye on > throughput limits etc. > > Feedback and comments are welcome, you can find Hermes documentation at: > http://hermes-pubsub.readthedocs.org/en/latest/index.html > > Hermes is published under Apache Licence: > https://github.com/allegro/hermes > > Best Regards, > Marcin >
[Announcement] Hermes - pub / sub broker built on top of Kafka
Hi Everyone Hermes is an asynchronous message broker built on top of Kafka. It provides reliable, fault tolerant REST interface for message publishing and adaptive push mechanisms for message sending. Hermes is used as a message broker for asynchronous communication between microservices. Some of the main features: - Performance and scalability - Hermes in production handling up to 30.000 msgs/sec with 99.9th percentile latency below 100 ms on a 4-node cluster. - Reliability - Hermes is used for publishing sensitive data, such as billing events, user account changes etc. Hermes allows to define more reliable policy for those important events - require acknowledge from all Kafka replicas and increase request timeouts. - Push model - It makes receiving messages from Hermes dead simple: you just have to write one HTTP endpoint in your service. It’s up to Hermes to create Kafka consumer, redeliver messages, keep eye on throughput limits etc. Feedback and comments are welcome, you can find Hermes documentation at: http://hermes-pubsub.readthedocs.org/en/latest/index.html Hermes is published under Apache Licence: https://github.com/allegro/hermes Best Regards, Marcin
