RE: Issue in retention with compact,delete cleanup policy

Bubbling this up to understand if anyone else are in similar use case. -Original Message- From: Koushik Chitta Sent: Sunday, February 23, 2020 1:35 PM To: users@kafka.apache.org; d...@kafka.apache.org Subject: [EXTERNAL] Issue in retention with compact,delete cleanup policy Hi, I

Issue in retention with compact,delete cleanup policy

Hi, I have a Topic with following config. cleanup.policy = compact,delete segment.bytes = 52428800 (~52 mb) min.compaction.lag.ms = 180 (30 min) delete.retention.ms = 8640 (1 day) retention.ms = 25920 (3 days) Ideally I would want the old records > 3 days to be deleted without

RE: Kakfa Broker scaling tips for high number of consumers for a single topic

What change did you observe in broker latency metrics 'totaltimems' ? -Original Message- From: aravind s Sent: Wednesday, November 13, 2019 11:03 PM To: users@kafka.apache.org Subject: Kakfa Broker scaling tips for high number of consumers for a single topic Hi, We have a use-case

RE: librdkafka : seek() to offset out of range

You can use query_watermark_offsets() to get high watermark of the topic partition to use as max offset. Regards, Koushik -Original Message- From: Aurelien DROISSART Sent: Thursday, August 29, 2019 5:32 AM To: users@kafka.apache.org Subject: librdkafka : seek() to offset out of range

RE: Partition Reassignment is getting stuck

The topic partition having the ISR issue might be on a offline directory. Look into the metric "offlineLogDirectoryCount" or use kafka-log-dirs.sh to understand the issue with that directory. In most cases, it would be the a KafkaStorage Exception. The partition reassignment would also be

RE: Curious case of under min ISR before offline partition

is too high to can not complete replication in time or there is record can not replicated to follower as some configs fo max size not proper? Best, Lisheng Koushik Chitta 于2019年8月29日周四 上午2:03写道： > Hi All, > > We had a topic partition(with 5 replication) going offline wh

Curious case of under min ISR before offline partition

Hi All, We had a topic partition(with 5 replication) going offline when leader of the partition was down. Below is some analysis Kafka server - 1.1 , relevant config (replica.fetch.wait.max.ms = 500, replica.fetch.min.bytes = 5, replica.lag.time.max.ms=1) Topic partition

Inconsistent Replica list for a partition

Hi, The no. of replicas of partition : 1 is less than rest of the partitions, In which scenarios this can happen ? Kafka version - 0.10.2 [cid:image001.png@01D4616F.405D2FA0] Thanks, Koushik

Re: Kafka on Windows

I've been using Kafka 0.10.x version on windows in production from past 1 year and plan to move to higher version soon . @M. Manna By Log cleaner functionality - Did you mean compaction in topics? I've not seen this issue. Did you file a bug on this? Can you share

RE: How many producers can publish to same topic concurrently?

We had ~75000 producers writing to a single topic on 10 broker cluster in Windows environment, yet to reach the max. limit considering the system resources used. I guess our bottleneck might be the no. of tcp connections to a broker. -Original Message- From: Pulkit Manchanda Sent:

RE: Kafka consumer to unzip stream of .gz files and read

You should read the message value as byte array rather than string . Other Approach is , while producing you can use the kafka compression = GZIP to have similar results. -Original Message- From: mayur shah Sent: Monday, May 21, 2018 1:50 AM To:

SSL Keymanager implementation using OS managed keystore

Hi, Running kafka on Windows OS, I would like to take advantage of windows managed keystore (eg: Windows-MY, can be loaded without a user, pass) .Eventually wants to use a custom keymanager implementation. >From the code I see, SSLFactory doesn't allow to use an existing keystore and >without