So can we roll segments more often? If the segments are small enough
probability of messages in a single segment reaching expiry will be higher.
However, will frequent roll-up of segments cause some side effects? Like
increased CPU, memory usage etc?
On Tue, May 29, 2018 at 11:52 PM Matthias J. Sax <matth...@confluent.io>
wrote:
> About the docs:
>
> Config `cleanup.policy` states:
>
> > A string that is either "delete" or "compact".
> > This string designates the retention policy to
> > use on old log segments. The default policy> ("delete") will discard old
> segments when their
> > retention time or size limit has been reached.> The "compact" setting
> will enable log
> > compaction on the topic.
>
> Because deletions happens based on segments, it it clear that some
> messages are retained longer, because a segment can only be dropped if
> _all_ messages in a segment passed the retention time.
>
> Does this make sense?
>
> Of course, we are always happy to improve the docs. Feel free to do a PR :)
>
>
> -Matthias
>
>
> On 5/29/18 3:01 AM, Shantanu Deshmukh wrote:
> > In one of my consumer application, I saw that 3 topics with 10 partitions
> > each were getting consumed by 5 different consumers having same consumer
> > group. And this application is seeing a lot of rebalances. Hence, I was
> > wondering about this.
> >
> > On Tue, May 29, 2018 at 1:57 PM M. Manna <manme...@gmail.com> wrote:
> >
> >> topic and consumer group have 1-to-many relationship. Each topic
> partition
> >> will have the messages guaranteed to be in order. Consumer rebalance
> issues
> >> can be adjusted based on the backoff and other params. What is exactly
> your
> >> concern regarding consumer group and rebalance?
> >>
> >>
> >>
> >> On 29 May 2018 at 08:26, Shantanu Deshmukh <shantanu...@gmail.com>
> wrote:
> >>
> >>> Hello,
> >>>
> >>> Is it wise to use a single consumer group for multiple consumers who
> >>> consume from many different topics? Can this lead to frequent rebalance
> >>> issues?
> >>>
> >>
> >
>
>
