Re: Initializing a consumer groups committed offset with -1

sue in another Kafka Exporter by forcibly setting > the lag to -1 so that I can monitor simple consumers, which don't > support discovery, that are not consuming yet from newly added > partitions: https://github.com/danielqsj/kafka_exporter/pull/17 > > > On Tue, 4 Feb 2020 at 20:10, Sean

Initializing a consumer groups committed offset with -1

Hi, I maintain Kafka Lag Exporter (an offset lag/latency monitoring tool). I use the Apache Kafka Java AdminClient and KafkaConsumer to retrieve offset information to calculate lag and other metrics, similar to how the kafka-consumer-groups.sh bin script works. A few users have reported Kafka

Re: [VOTE] 2.4.0 RC1

Hi, I also used Eric's test script. I had a few issues running it that I address below[0][1], otherwise looks good. - Signing keys all good - All md5, sha1sums and sha512sums are good - A couple transient test failures that passed on a second run

Re: Producer send blocking when destination partition does not exist

Mon, Nov 4, 2019 at 1:37 PM Jamie wrote: > Hi Sean, > Out of interest, is auto topic creation enabled on the brokers? > Thanks, > Jamie > > > -Original Message- > From: Sean Glover > To: users > Sent: Mon, Nov 4, 2019 04:21 PM > Subject: Producer send blo

Producer send blocking when destination partition does not exist

r clientId=producer-1] Exception occurred during message send: org.apache.kafka.common.errors.TimeoutException: Topic topic-1-1 not present in metadata after 6 ms. -- Sean Glover Principal Engineer, Alpakka, Lightbend, Inc. <https://lightbend.com> @seg1o <https://twitter.com/seg1o>,

Re: Questions about Producer per Task/Partition in Streams EoS Impl.

throughput; 2) on the broker side, increased num.connections > and request rate (i.e. more smaller requests compared to fewer larger > requests) is also an increased load overhead. > > We plan to address KIP-447 in the near term so that we can lift these > performance hurdles. >

Questions about Producer per Task/Partition in Streams EoS Impl.

Hi, I would like to understand better how a `KafkaProducer` is used within Kafka Streams EoS use cases. In the Streams Exactly Once Design [1] document it states that there is one producer per StreamThread: Each thread contains one producer client and two consumer clients (one for > normal

Re: SAM Scala aggregate

Hi, SAM conversions can be enabled in 2.11 with the -Xexperimental to scalac. However, this version of SAM conversions isn't recommended for production and was significantly refactored for 2.12. When we contributed this API to Kafka we explicitly removed the SAM conversions because we didn't

Re: Kafka in virtualized environments

Giresh, I'm curious what your solution was. Did you use locally attached storage for your ZK ensemble? Did you move it to static machines? On Thu, Nov 30, 2017 at 4:50 PM, John Yost wrote: > Great point by Girish--its the delays of syncing with Zookeeper that are >

Re: Using Kafka on DC/OS + Marathon

No, I don't. I help others that do :) On Tue, Oct 3, 2017 at 1:12 PM, Valentin Forst <valen...@aseno.de> wrote: > Hi Sean, > > Thanks a lot for this info ! > Are you running DC/OS in prod? > > Regards > Valentin > > > Am 03.10.2017 um 15:29 schrieb Se

Re: Using Kafka on DC/OS + Marathon

Hi Valentin, Kafka is available on DC/OS in the Catalog (aka Universe) as part of the `kafka` package. Mesosphere has put a lot of effort into making Kafka work on DC/OS. Since Kafka requires persistent disk it's required to make sure after initial deployment brokers stay put on their assigned

Re: How to clear a particular partition?

Alternatively you can set topic overrides for retention.bytes. By turning back file.delete.delay.ms that change should be nearly instant after the next log cleanup cycle. # Apply topic config override $ kafka-configs --alter --entity-type topics --entity-name test --zookeeper localhost:32181