Re: How should I properly create the consumer group and reset the offset from all topics in kakfa CLI?
Hi! Did consumer group groupA exist before the --reset-offsets operation was performed? Thanks, Kirk On Fri, May 5, 2023, at 3:18 AM, choi ewain wrote: > Hello All. > > I am a newbie learning kakfa. > > I have 1000+topics in my kakfa, each topic contains 14 days of data, about > 120,000 messages. > > I created a new consumer group: groupA, and executed the command to reset the > offset. > > sh kafka-consumer-groups.sh --reset-offsets --execute --to-datetime > "2023-05-03T00:00:00.000" --group groupA --bootstrap-server kfk-data- > 001:9092,kfk-data-002:9092,kfk-data-003:9092 --all-topics > > But it didn't work as I expected, I tried to use groupA pull data in Java, > and it still consumes the starting data for each topics. > > How should I properly create the group and reset the offset from all topics? > > Thanks. > > Ewain Choi. > >
Re: Consumer has high network bandwidth when consuming no messages
Hi Lee, Some questions: 1. Can you enable metrics for your consumer(s)? See [1] and [2] for links on consumer-level and fetch-level metrics, respectively. 2. Can you enable more detailed logging? 3. What do you mean specifically by “concurrency of 2?” 4. Can you verify the name of the fetch.wait.max.ms configuration? I don’t see that configuration option. Is that a broker configuration or a consumer configuration? Thanks, Kirk [1] https://kafka.apache.org/documentation/#consumer_group_monitoring [2] https://kafka.apache.org/documentation/#consumer_fetch_monitoring > On Apr 13, 2023, at 4:52 AM, Lee Carroll > wrote: > > Hi All, > I've a spring boot kafka consumer group, consuming a topic with 30 > partitions with a concurrency of 2 (staging set up). > > The network traffic in and out was very high >200k per second even when > consuming no messages. > > I've adjusted fetch.wait.max.ms to be significantly higher (15,000) than > the default which has had a significant impact but the network is still > seeing ~30k a second. > > Is this to be expected? > > Cheers Lee C
Re: [ANNOUNCE] Apache Kafka 3.2.1
Thanks for driving this, David! On Mon, Aug 1, 2022, at 4:45 PM, David Arthur wrote: > The Apache Kafka community is pleased to announce the release for > Apache Kafka 3.2.1 > > This is a bugfix release with several fixes since the release of > 3.2.0. A few of the major issues include: > > * KAFKA-14062 OAuth client token refresh fails with SASL extensions > * KAFKA-14079 Memory leak in connectors using errors.tolerance=all > * KAFKA-14024 Cooperative rebalance regression causing clients to get stuck > > > All of the changes in this release can be found in the release notes: > > https://www.apache.org/dist/kafka/3.2.1/RELEASE_NOTES.html > > > You can download the source and binary release (Scala 2.12 and 2.13) from: > > https://kafka.apache.org/downloads#3.2.1 > > --- > > > Apache Kafka is a distributed streaming platform with four core APIs: > > ** The Producer API allows an application to publish a stream of > records to one or more Kafka topics. > > ** The Consumer API allows an application to subscribe to one or more > topics and process the stream of records produced to them. > > ** The Streams API allows an application to act as a stream processor, > consuming an input stream from one or more topics and producing an > output stream to one or more output topics, effectively transforming > the input streams to output streams. > > ** The Connector API allows building and running reusable producers or > consumers that connect Kafka topics to existing applications or data > systems. For example, a connector to a relational database might > capture every change to a table. > > > With these APIs, Kafka can be used for two broad classes of application: > > ** Building real-time streaming data pipelines that reliably get data > between systems or applications. > > ** Building real-time streaming applications that transform or react > to the streams of data. > > > Apache Kafka is in use at large and small companies worldwide, > including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, > Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and > Zalando, among others. > > A big thank you for the following 19 contributors to this release! > > Akhilesh Chaganti, Bruno Cadonna, Christopher L. Shannon, David > Arthur, Divij Vaidya, Eugene Tolbakov, Guozhang Wang, Ismael Juma, > James Hughes, Jason Gustafson, Kirk True, Lucas Bradstreet, Luke Chen, > Nicolas Guyomar, Niket Goel, Okada Haruki, Shawn Wang, Viktor > Somogyi-Vass, Walker Carlson > > We welcome your help and feedback. For more information on how to > report problems, and to get involved, visit the project website at > https://kafka.apache.org/ > > > Thank you! > > Regards, > David Arthur >
Re: [ANNOUNCE] New committer: Luke Chen
Congrats, Luke! On Wed, Feb 9, 2022, at 3:22 PM, Guozhang Wang wrote: > The PMC for Apache Kafka has invited Luke Chen (showuon) as a committer and > we are pleased to announce that he has accepted! > > Luke has been actively contributing to Kafka since early 2020. He has > made more than 120 commits on various components of Kafka, with notable > contributions to the rebalance protocol in Consumer and Streams (KIP-766, > KIP-726, KIP-591, KAFKA-12675 and KAFKA12464, to just name a few), as well > as making an impact on improving test stability of the project. Aside from > all his code contributions, Luke has been a great participant in > discussions across the board, a very active and helpful reviewer of other > contributors' works, all of which are super valuable and highly appreciated > by the community. > > > Thanks for all of your contributions Luke. Congratulations! > > -- Guozhang, on behalf of the Apache Kafka PMC >
Re: Memory leak with kafka clients 3.0.0
Hi Willian, I don't see any recent Jiras that seem to cover this. I noticed two requests in the linked spring-kafka GitHub thread: 1. File a Jira on the Kafka project with relevant configuration, etc. 2. Provide a stripped-down, reproducible test case using Kafka (sans Spring) Having those two bits of information is a good first step to solving the mystery. I'm sure the rest of the community would be very keen to fix any memory usage regressions. Thanks, Kirk On Wed, Jan 12, 2022, at 2:08 AM, Willian Dallastella wrote: > Hi, > > we are using Spring boot with *spring-kafka* and when updating to Spring > Boot 2.6+ we have memory leak in the kafka producer as it comes with > *kafka-cients > 3.0.0*. > > After testing, using Spring Boot 2.6 and just *downgrading kafka-clients to > 2.7.1*, the *memory leak stopped*. > > Here are more details: > https://github.com/spring-projects/spring-kafka/issues/2056 > > Are you aware about this issue? > > Best regards, > Willian Dallastella >
