kafka message expiry question

hi, as far as i understand, log retention time in kafka will delete message that's older than the retention time. i'm wonder what it implies for consumer since i'm using simple consumer to manage offsets in predefined consumer group. say i have a list of messages for a partition of topic: 1,2,3,

Re: ISR shrink to 0?

In that case, no leader will be elected until at least 1 replica in ISR comes back. Thanks, Jun On Wed, Nov 19, 2014 at 6:58 PM, Jason Rosenberg wrote: > What if it never comes back with unclean leader election disabled (but > another broker does come back)? > > On Wed, Nov 19, 2014 at 9:32 PM

Re: ISR shrink to 0?

What if it never comes back with unclean leader election disabled (but another broker does come back)? On Wed, Nov 19, 2014 at 9:32 PM, Jun Rao wrote: > In that case, we just wait until the broker in ISR is back and make it the > leader and take whatever data is has. > > Thanks, > > Jun > > On T

Re: Aeron, a high throughput, low latency messaging system

Here's a good writeup that also mentions Kafka: http://highscalability.com/blog/2014/11/17/aeron-do-we-really-need-another-messaging-system.html Comparing it to Kafka is an apples to oranges comparison from what I can tell. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Mana

Re: Drawbacks to setting log.roll.hours=1 ?

Hello, One thing you need to check is that the file handler limit on your machine is set high enough for the number of files on disk plus sockets. Other than that, as long as you are consuming at the tail of the log most of the time it should be fine. Guozhang On Wed, Nov 19, 2014 at 9:23 AM, Th

Re: ISR shrink to 0?

In that case, we just wait until the broker in ISR is back and make it the leader and take whatever data is has. Thanks, Jun On Tue, Nov 18, 2014 at 10:36 PM, Jason Rosenberg wrote: > Ok, > > Makes sense. But if the node is not actually healthy (and underwent a hard > crash) it would likely n

Re: Aeron, a high throughput, low latency messaging system

Hi Roger While Aeron does indeed prioritise latency, it also has exceptional throughput performance as well. The early benchmark one user did put it at 2x Kafka: https://groups.google.com/d/msg/mechanical-sympathy/GrEce0gP7RU/gDzl5JG2dfwJ. While benchmarking is fraught with perils and should be ta

Re: will adding partitions to existing topic change leadership to existing partitions?

Hi Shlomi, By just use "kafka-topics.sh --zookeeper localhost:2181 --alter --topic test_topic --partitions 8" the controller will auto assign replicas to the newly added partitions, but will not touch the existing ones. Guozhang On Mon, Nov 17, 2014 at 11:13 PM, Shlomi Hazan wrote: > Hi Guozha

Re: Aeron, a high throughput, low latency messaging system

Hi Daniel, Thanks for sharing this. Looks like a great project. I probably don't know enough to give a great answer but will throw in my 2c anyway. I think Kafka prioritizes throughput and aeron prioritizes latency. As you mentioned, maybe Aeron could replace the current Kafka TCP protocol. T

Drawbacks to setting log.roll.hours=1 ?

Hello, We have a 5 node cluster, each with a 5 disk JBOD storage running Kafka 0.8.1. We are interested in the possibility of using the OffsetRequest API taking a time to get offsets from a given time. Because that API can only "find" times where the log segments roll, we would need to set log.

Re: Topic Creation

We have a ticket open for making admin tools like topic creation to be sync: KAFKA-1125 On Tue, Nov 18, 2014 at 9:25 PM, Jun Rao wrote: > Yes, that's the best option for now. The creation is async. > > Thanks, > > Jun > > On Tue, Nov 18, 2014 at