Announcing the Community Over Code 2025 Streaming Track

Hi all, Community Over Code , the ASF conference, will be held in Minneapolis, Minnesota, September 11-14, 2025. The call for presentations is open now through April 21, 2025. I am one of the co-chairs for th

Can I tell mirrormaker2 to start (re)consuming from a specific offset?

ld, I could manually change offsets by doing this kafka-consumer-groups.sh --bootstrap-server kafka:9092 --group groupname --reset-offsets --to-latest--execute Is there any similar way to manipulate mirrormaker2, to tell it to start (re)consuming from a specific offset in a source topic? Thanks, -James

Re: SmallRye bootstrap.servers value no longer accepted in kafka-clients v3.8.0

Looking at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(List, ClientDnsLookup) shows that error message is referring to a singular url from the list. So the issue is earlier in the chain - the list is not being decoded from the CSV. > On 12 Sep 2024, at 03:03, Josep Prat wr

Announcing the Community Over Code 2024 Streaming Track

Hi all, Community Over Code , the ASF conference, will be held in Denver, Colorado, October 7-10, 2024. The call for presentations is open now

Announcing the Community Over Code 2023 Streaming Track

Hi all, Community Over Code , the ASF conference, will be held in Halifax, Nova Scotia October 7-10, 2023. The call for presentations is open now through July 13, 2023. I am one of the co-chairs for the stream

JOB | Platform Lead (Singapore or London)

ke to discuss I can be reached using "JamesBTobin (at) Gmail (dot) Com". Kind regards, James

KIP-794 inhibits Partitioner delegation with no fallback option

Prior to KIP-794 it was possible to create a custom Partitioner that could delegate to the DefaultPartitioner. This has been deprecated so we can now only delegate to BuiltInPartitioner.partitionForKey which does not handle a non-keyed message. Hence there is now no mechanism for a custom Part

Re: Apache Kafka :-Producer-Consumer at different network

Happy to help, Namita. Regards, Jim Ziesig On Fri, Sep 30, 2022 at 7:51 AM Namita Jaokar wrote: > Hi James, > > Thanks alot James for the solution provided. The below configurations did > work when producer and consumer applications were checked with the > scenarios mentione

Re: Apache Kafka :-Producer-Consumer at different network

and hostname to /etc/hosts. I believe your producer is on Windows so it might be best to stick with the IP address. Good luck, Jim Ziesig On Wed, Sep 28, 2022 at 8:21 AM Namita Jaokar wrote: > Hi James, > > As suggested, I did try below steps but still my producer is unable to >

Re: Apache Kafka :-Producer-Consumer at different network

Hi Namita, This is likely a listener configuration issue. The bootstrap server setting only gets you so far, your client must be able to contact the server via a listener address. I have seen this occur on a few systems where a hostname could not be identified by Kafka, so the server advertises

Re: mTLS client hostname validation is not working

tup a bit better. > > Kind regards, > > > Richard Bosch > > Developer Advocate > > Axual BV > > https://axual.com/ > > > On Thu, Sep 15, 2022 at 9:55 PM James Ziesig > wrote: > > > Hi, > > > > I have configured mTLS on a three serv

mTLS client hostname validation is not working

Hi, I have configured mTLS on a three server Kafka cluster. The servers and clients are all communicating properly, except I am having trouble with client hostname validation when the client is using a cert from a different host. I would expect this to fail on handshake like it does when the cer

Re: [ANNOUNCE] Apache Kafka 3.2.0

https://cwiki.apache.org/confluence/display/KAFKA/KIP-791%3A+Add+Record+Metadata+to+State+Store+Context> > adds method recordMetadata() to the StateStoreContext, I know that the page has already been published, but should we fix that typo? Thanks! -James > On May 17, 2022, at 9:01 AM, Brun

Re: Unexpected loss of Offsets

ther issues, or the 2.8.2 if that's not possible. What can we do to make this happen ASAP? Regards, James. On 29/04/2022, at 14:50, Luke Chen mailto:show...@gmail.com>> wrote: Hi James, So far, v2.8.2 is not planned, yet. And usually, the patch release only has one, that is, v2.

Re: Unexpected loss of Offsets

Luke, Do you know if 2.8.2 will be released anytime soon? It appears to be waiting on https://issues.apache.org/jira/browse/KAFKA-13805 for which fixes are available. Regards, James. On 11/04/2022, at 14:22, Luke Chen mailto:show...@gmail.com>> wrote: Hi James, This looks like this

Unexpected loss of Offsets

utes - not using static assignment). Is it possible the logic is actually checking for no consumer now and no offsets for 7 days instead? Server and Client are 2.7.2. Sorry I don't have any more detailed server-side logs. Regards, James.

Re: Consumer failure after rolling Broker upgrade

not occur with 2.5.1 and 2.7.0 Clients. This make me suspect that https://issues.apache.org/jira/browse/KAFKA-10793 introduced this issue. Regards, James. On 24/11/2021, at 14:35, James Olsen mailto:ja...@inaseq.com>> wrote: Luke, We did not upgrade to resolve the issue. We simply

Re: Consumer failure after rolling Broker upgrade

Luke, We did not upgrade to resolve the issue. We simply restarted the failing clients. Regards, James. On 23/11/2021, at 16:10, Luke Chen mailto:show...@gmail.com>> wrote: Hi James, > Bouncing the clients resolved the issue Could you please describe which version you upgrade to, t

Consumer failure after rolling Broker upgrade

atience to keep reading this far then maybe you do! Bouncing the clients resolved the issue, but I'd be interested if any experts out there can identify if there is any weakness in the 2.7.1 version. Regards, James.

Re: How is __consumer_offsets partitioned?

If it's of any value to you, we use the following test to check that we have a well balanced set of consumer group ids. Note that in the code, ConsumerGroups.ALL_GROUPS is simply a list of all our consumer group ids. Spreading the offset commit load across these partitions evenly helps in lev

Re: [ANNOUNCE] Apache Kafka 2.6.1

Thank you Mickael for running the release. Good job everyone! -James Sent from my iPhone > On Jan 11, 2021, at 2:17 PM, Mickael Maison wrote: > > The Apache Kafka community is pleased to announce the release for > Apache Kafka 2.6.1. > > This is a bug fix release and it

Re: [ANNOUNCE] New committer: David Jacot

Congratulations, David! -James > On Oct 16, 2020, at 9:01 AM, Gwen Shapira wrote: > > The PMC for Apache Kafka has invited David Jacot as a committer, and > we are excited to say that he accepted! > > David Jacot has been contributing to Apache Kafka since July 2015 (!) &

Re: [ANNOUNCE] New committer: Chia-Ping Tsai

Congratulations Chia-Ping! -James > On Oct 19, 2020, at 10:24 AM, Guozhang Wang wrote: > > Hello all, > > I'm happy to announce that Chia-Ping Tsai has accepted his invitation to > become an Apache Kafka committer. > > Chia-Ping has been contributing to Kafka sin

Re: [ANNOUNCE] New committer: A. Sophie Blee-Goldman

Congratulations, Sophie! -James > On Oct 19, 2020, at 9:40 AM, Matthias J. Sax wrote: > > Hi all, > > I am excited to announce that A. Sophie Blee-Goldman has accepted her > invitation to become an Apache Kafka committer. > > Sophie is actively contributing to Kafk

Re: MM2 max.request.size setting

Even with the errors complaining, it does appear to have worked. Thanks much! James Lavoy Senior Infrastructure Engineer Verizon - Protectwise 1601 Wewatta Street #700 Denver, Co > On Aug 31, 2020, at 2:16 PM, nitin agarwal wrote: > > I am not sure why it is picked in ConsumerCo

Re: MM2 max.request.size setting

ng those values. James Lavoy Senior Infrastructure Engineer Verizon - Protectwise 1601 Wewatta Street #700 Denver, Co > On Aug 31, 2020, at 1:08 PM, nitin agarwal wrote: > > If you have clusters like DC1 and DC2 then you can add the following > configuration: > DC1->DC2.p

Re: MM2 max.request.size setting

Through the connect-mirror-maker.sh helper script. James Lavoy Senior Infrastructure Engineer Verizon - Protectwise 1601 Wewatta Street #700 Denver, Co > On Aug 31, 2020, at 1:04 PM, nitin agarwal wrote: > > How are you running MM2 ? > > Thanks, > Nitin > > On Mo

MM2 max.request.size setting

s the specific value I'm talking about: [2020-08-31 15:42:17,066] INFO ProducerConfig values: max.request.size = 1048576 Does anybody know off-hand the right config line to override this limit? Thank you for your time and attention. James Lavoy Senior Infrastructure Engineer Verizon

Re: Frequent consumer offset commit failures

;ll probably implement that ourselves and also incorporate an equivalent to auto.commit.interval.ms. Regards, James. On 20/06/2020, at 02:15, Ricardo Ferreira mailto:rifer...@riferrei.com>> wrote: James, If I were you I would start investigating what is causing this network drops between your cluster an

Frequent consumer offset commit failures

AndMetadata{offset=514, leaderEpoch=2, metadata=''}} failed: org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing the latest consumed offsets. Caused by: org.apache.kafka.common.errors.DisconnectException ... plus many more with increasing 50=OffsetAndMetadata{offset=xxx ...} (the busy Partition). Thanks, James.

Consumer Group got stuck in CompletingRebalance

4-13 21:37:20,308] INFO [GroupCoordinator 6]: Stabilized group trackers-etl generation 4987 (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator) Has anyone seen this before? Should I file a JIRA ticket? Was there a better process than restarting the broker? -- James Brown Systems Engineer

Re: acks this category properties in Which *.java Defined???

org.apache.kafka.clients.producer.ProducerConfig org.apache.kafka.clients.consumer.ConsumerConfig > On 3/04/2020, at 04:30, 一直以来 <279377...@qq.com> wrote: > > Properties props = new Properties(); props.put("bootstrap.servers", > "localhost:9092"); props.put("acks", "all"); props.put("key.seri

Re: Reg : Slowness in Kafka

ven’t raised an issue for it as I would have no expectation of it being fixed. I just made my client and server versions match and implemented a workaround for the issue I wanted the 2.4.0 client to resolve. Regards, James. On 31/03/2020, at 12:16, JALADY Christophe mailto:christophe.jal...@lapos

Re: Problems when Consuming from multiple Partitions

Resolved by downgrading Client to 2.2.2 and implementing an application level heartbeat on every Producer to avoid he UNKNOWN_PRODUCER_ID issue. > On 9/03/2020, at 16:08, James Olsen wrote: > > P.S. I guess the big question is what is the best way to handle or avoid > UNKNOWN_PROD

Re: Reg : Slowness in Kafka

Also check your Kafka Client and Server versions. There are serious latency issues when mixing different client and server versions IF your consumers handle multiple partitions. > On 27/03/2020, at 12:59, Chris Larsen wrote: > > Hi Vidhya, > > How many tasks are you running against the topic

Re: Problems when Consuming from multiple Partitions

P.S. I guess the big question is what is the best way to handle or avoid UNKNOWN_PRODUCER_ID when running versions that don’t include KAFKA-7190 / KAFKA-8710 ? We are using non-transactional idempotent Producers. > On 9/03/2020, at 12:59 PM, James Olsen wrote: > > For completene

Re: Problems when Consuming from multiple Partitions

? We can choose 2.2.1 or 2.3.1 for the Broker (AWS recommend 2.2.1 although don’t state why). Based on the experiences below, I would then go with the corresponding 2.2.2 or 2.3.1 Client version. Which combo would people recommend? > On 9/03/2020, at 12:03 PM, James Olsen wrote: > &

Re: Problems when Consuming from multiple Partitions

wanted the fix for https://issues.apache.org/jira/browse/KAFKA-7190 as we may have some very quiet Topics.2.3.x does have some handling for this as implied by https://issues.apache.org/jira/browse/KAFKA-8483 but I’m not sure it is as complete. Regards, James. On 9/03/2020, at 11:54 AM, Jamie

Re: Problems when Consuming from multiple Partitions

4,889 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher] 'EE-ManagedThreadFactory-default-Thread-2' [Consumer clientId=consumer-LedgerService-group-1, groupId=LedgerService-group] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(Ledger-1), toForget=(), implied=(Ledg

Re: Problems when Consuming from multiple Partitions

up-1, groupId=Redacted-group] Fetch READ_UNCOMMITTED at offset 40 for partition MyTopic-1 returned fetch data (error=NONE, highWaterMark=41, lastStableOffset = 41, logStartOffset = 0, preferredReadReplica = absent, abortedTransactions = null, recordsSizeInBytes=573) On 5/03/2020, at 11:45 PM

Problems when Consuming from multiple Partitions

I’m seeing behaviour that I don’t understand when I have Consumers fetching from multiple Partitions from the same Topic. There are two different conditions arising: 1. A subset of the Partitions allocated to a given Consumer not being consumed at all. The Consumer appears healthy, the Thread

Re: Significant CPU utilization increase in 2.4.0

I filed https://issues.apache.org/jira/browse/KAFKA-9339 On Fri, Dec 27, 2019 at 12:24 PM Ismael Juma wrote: > Hi James, > > Thanks for the report. This is not expected. How many partitions per broker > and what Java version? I suggest filing a JIRA and, if possible, attach a >

Significant CPU utilization increase in 2.4.0

constant or worse when we upgrade something with more load on it? -- James Brown Engineer

New topic gets stuck in bad state after Zookeeper error

oker 6 and as soon as it came up, it assumed leadership of the partition and everything started working fine. Has anyone else seen this behavior before? The fact that a partition was unavailable but the mbean showed 0 under-replicated and 0 un-available topics is extremely concerning to me. -- James Brown Systems Engineer

Re: please care and vote for Chinese people under cruel autocracy of CCP, great thanks!

Why is this group being targeted by the false political propaganda? The sender did not even use a valid email account. On Thu, Aug 29, 2019 at 7:05 AM ant_fighter wrote: > Hi all, > Sorry for disturbing you guys. Though I don't think here as a proper place > to do this, I need your help, your vo

Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax

Congrats!! -James Sent from my iPhone > On Apr 18, 2019, at 2:35 PM, Guozhang Wang wrote: > > Hello Everyone, > > I'm glad to announce that Matthias J. Sax is now a member of Kafka PMC. > > Matthias has been a committer since Jan. 2018, and since then he conti

Re: Proxying the Kafka protocol

resee any fundamental issues with this approach? James Grant Developer - Expedia Group On Tue, 19 Mar 2019 at 16:13, Hans Jespersen wrote: > > > You might want to take a look at kafka-proxy ( see > https://github.com/grepplabs/kafka-proxy < > https://github.com/grepplabs/kafka

Proxying the Kafka protocol

anybody know of any reason it would be impossible (or a bad idea) to do? James Grant Developer - Expedia Group

Re: [ANNOUNCE] New Committer: Randall Hauch

Congrats, Randall! Well deserved! -James Sent from my iPhone > On Feb 14, 2019, at 6:16 PM, Guozhang Wang wrote: > > Hello all, > > The PMC of Apache Kafka is happy to announce another new committer joining > the project today: we have invited Randall Hauch as a project

Re: [ANNOUNCE] New Committer: Vahid Hashemian

Congrats, Vahid!! -James > On Jan 15, 2019, at 2:44 PM, Jason Gustafson wrote: > > Hi All, > > The PMC for Apache Kafka has invited Vahid Hashemian as a project committer > and > we are > pleased to announce that he has accepted! > > Vahid has made numer

Re: [ANNOUNCE] Apache Kafka 2.1.0

Thanks Dong for running the release, and congrats to everyone in the community! -James Sent from my iPhone > On Nov 21, 2018, at 10:09 AM, Dong Lin wrote: > > The Apache Kafka community is pleased to announce the release for Apache > Kafka 2.1.0 > > > This is a major

Re: Kafka on Eclipse Setup

Matthias, Thanks for your suggestion but it did not work. I finally have to remove all reference of test libraries and just use main. I was able to build all libraries without any errors. James > On Oct 15, 2018, at 12:45 PM, Matthias J. Sax wrote: > > You can try to

Re: Kafka on Eclipse Setup

stream codes. I can manually copy all Kafka related jar files but is this the right way? I thought after I download the codes from Github, everything should work. James > On Oct 13, 2018, at 10:08 AM, Vahid Hashemian > wrote: > > This article was written a while back but stil

Kafka on Eclipse Setup

or path not defined messages. I don’t know if I need to fix them one by one or if there is any jar files that I need to include. James

Re: [ANNOUNCE] New committer: Colin McCabe

Congrats, Colin! -James > On Sep 25, 2018, at 1:39 AM, Ismael Juma wrote: > > Hi all, > > The PMC for Apache Kafka has invited Colin McCabe as a committer and we are > pleased to announce that he has accepted! > > Colin has contributed 101 commits and 8 KI

Re: unable to start kafka when zookeeper cluster is in working but unhealthy state

resolved. Or does it also work for this case? Thanks for your responses. This is a UTF-8 formatted mail --- James C.-C.Yu +886988713275 +8615618429976 2018-09-26 19:36 GMT+08:00 Manikumar : > You can try using Kafka 2.0 release. Original issue

unable to start kafka when zookeeper cluster is in working but unhealthy state

r) I thought we can start a kafka as long as the zookeeper cluster is working.regardless there is one zookeeper node (3 in total) fails to run? This is a UTF-8 formatted mail --- James C.-C.Yu +886988713275 +8615618429976

Re: Need info

MySQL and PostgrepSQL. > On Sep 12, 2018, at 12:34 PM, Mich Talebzadeh > wrote: > > Hi James, > > As a matter of interest this streaming data is fed into some Operational > Data Store ODS) like MongoDB? > > In general using this method will create a near real time sn

Re: Need info

We have banking customers sending data from DB2 z to Kafka linux (not cloud) with transaction rate 30K per seconds. Kafka can handle more than this rate. > On Sep 12, 2018, at 2:31 AM, Chanchal Chatterji > wrote: > > Hi, > > In the process of mainframe modernization, we are attempting to str

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

Congrats Dong! -James > On Aug 20, 2018, at 3:54 AM, Ismael Juma wrote: > > Hi everyone, > > Dong Lin became a committer in March 2018. Since then, he has remained > active in the community and contributed a number of patches, reviewed > several pull requests and partici

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Congrats and great job, everyone! Thanks Rajini for driving the release! -James Sent from my iPhone > On Jul 30, 2018, at 3:25 AM, Rajini Sivaram wrote: > > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 2.0.0. > > > > >

Does anyone publish or subscribe kafka topic in TestNG?

cord(topicName, Integer.toString(i), Integer.toString(i))); } System.out.println("Message sent successfully"); producer.close(); } } This is a UTF-8 formatted mail --- James C.-C.Yu +886988713275 +8615618429976

org.apache.kafka.common.errors.UnsupportedVersionException in Java

erContainerFactory factory = new ConcurrentKafkaListenerContainerFactory<>(); factory.setConsumerFactory(consumerFactory()); factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE); return factory; } Thank you. This is a UTF-8 formatted mail --- James C.-C.Yu +886988713275 +8615618429976

Re: [ANNOUNCE] Apache Kafka 1.1.0 Released

Thanks Damian and Rajini for running the release! Congrats and good job everyone! -James Sent from my iPhone > On Mar 29, 2018, at 2:27 AM, Rajini Sivaram wrote: > > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 1.1.0. > > >

Re: [ANNOUNCE] New Committer: Dong Lin

Congrats, Dong! -James > On Mar 28, 2018, at 10:58 AM, Becket Qin wrote: > > Hello everyone, > > The PMC of Apache Kafka is pleased to announce that Dong Lin has accepted > our invitation to be a new Kafka committer. > > Dong started working on Kafka about four years

Re: [ANNOUNCE] Apache Kafka 1.0.1 Released

Congrats, everyone! Thanks for driving the release, Ewen! -James > On Mar 6, 2018, at 1:22 PM, Guozhang Wang wrote: > > Ewen, thanks for driving the release!! > > > Guozhang > > On Tue, Mar 6, 2018 at 1:14 PM, Ewen Cheslack-Postava wrote: > >> The Apa

Consume latest offset after rebalance

ay to catch the "NOT_COORDINATOR_FOR_GROUP" error I could manually alter the offset? I'm currently using 0.9.1. Thanks -James

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congrats Rajini! -James Sent from my iPhone > On Jan 17, 2018, at 10:48 AM, Gwen Shapira wrote: > > Dear Kafka Developers, Users and Fans, > > Rajini Sivaram became a committer in April 2017. Since then, she remained > active in the community and contributed major patche

Re: [VOTE] KIP-247: Add public test utils for Kafka Streams

+1 (non-binding) -James Sent from my iPhone > On Jan 17, 2018, at 6:09 PM, Matthias J. Sax wrote: > > Hi, > > I would like to start the vote for KIP-247: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams > > > -Matthias >

what are common ways to convert info on a web site into a log entry?

the time when it is likely to only do a few seconds of work each month? Many thanks, James.

Re: what are common ways to convert info on a web site into a log entry?

thanks! I will write my reply into the mail group once I have fiddled with your suggestions. James. > On Jan 15, 2018, at 2:20 PM, Jacob Sheck wrote: > > You will need to find or create something to accomplish this. Topics in > Kafka primarily act as queues. If you search the

what are common ways to convert info on a web site into a log entry?

the time when it is likely to only do a few seconds of work each month? Many thanks, James.

Re: [ANNOUNCE] New committer: Matthias J. Sax

Congrats, Matthias!! Well deserved! -James > On Jan 12, 2018, at 2:59 PM, Guozhang Wang wrote: > > Hello everyone, > > The PMC of Apache Kafka is pleased to announce Matthias J. Sax as our > newest Kafka committer. > > Matthias has made tremendous contributions to

Re: Insanely long recovery time with Kafka 0.11.0.2

hen I used strace to see what I/O the broker was doing, it was reading ALL the segments for the partitions. -James > On Jan 11, 2018, at 10:56 AM, Vincent Rischmann wrote: > > If anyone else has any idea, I'd love to hear it. > > Meanwhile, I'll resume upgrading my bro

JOB | Permanent Sysadmin (Amsterdam, Netherlands OR remote)

using "JamesBTobin (at) Gmail (dot) Com". Kind regards, James

Re: [ANNOUNCE] New committer: Onur Karaman

Congrats Onur! Well deserved! -James > On Nov 6, 2017, at 9:24 AM, Jun Rao wrote: > > Hi, everyone, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer Onur > Karaman. > > Onur's most significant work is the improvement of Kafka controll

Re: [ANNOUNCE] Apache Kafka 1.0.0 Released

Good job, everyone! And thanks Guozhang for running the release! -James > On Nov 1, 2017, at 7:27 AM, Guozhang Wang wrote: > > The Apache Kafka community is pleased to announce the release for Apache > Kafka 1.0.0. > > This is a major release of the Kafka project, and is n

How do I instantiate a metrics reporter in Kafka Streams, with custom config?

on-recognized keys in the properties object? And actually, for the metrics.reporter class itself, am I supposed to specify it as: metrics.reporter or metric.reporters producer.metric.reporters consumer.metric.reporters Thanks, -James

JOB | Permanent Remote Cassandra Database Engineer

om". Kind regards, James

Re: In which scenarios would "INVALID_REQUEST" be returned for "Offset Request"

Your client library might be sending a message that is too old or too new for your broker to understand. What version is your Kafka client library, and what version is your broker? -James Sent from my iPhone > On Sep 22, 2017, at 4:09 PM, Vignesh wrote: > > Hi, > > In which

Re: Metrics: committed offset, client version

188+-+Add+new+metrics+to+support+health+checks> -James > On Sep 20, 2017, at 2:37 AM, Stas Chizhov wrote: > > Hi! > > I am wondering if there are broker/client metrics for: > - client version (to keep track of clients that needs an upgrade) > - committed offsets (to det

Re: Kafka Internals Video/Blog

e soon. -James > On Sep 20, 2017, at 9:49 AM, Raghav wrote: > > Hi > > Just wondering if there is any video/blog that goes over Kafka Internal and > under the hood design and implementation details. I am a newbie and I would > like to dabble with the code and understand desi

Re: Improving Kafka State Store performance

.level="debug", you can see a bunch of metrics around the state stores. Stuff like put-latency-avg, for example. See http://kafka.apache.org/documentation/#kafka_streams_store_monitoring <http://kafka.apache.org/documentation/#kafka_streams_store_monitoring> -James > On Sep 16, 20

Re: Consumer group metadata retention

The offsets.retention.minutes value (1440 = 24 hours = 1 day) is a broker level configuration, and can't be changed dynamically during runtime. You would have to modify the broker configurations, and restart the brokers. -James > On Jul 25, 2017, at 9:43 PM, Raghu Angadi wrote: &

Re: Tuning up mirror maker for high thruput

tuning affect those consumers that are close to the source datacenter? Will they continue to function well? -James > On Jul 23, 2017, at 7:16 AM, Todd Palino wrote: > > One of the best pieces of advice I can offer is that you really need to run > the mirror maker in the same phys

Re: Tuning up mirror maker for high thruput

video at https://youtu.be/oQe7PpDDdzA -James Sent from my iPhone > On Jul 21, 2017, at 9:25 AM, Sunil Parmar wrote: > > We're trying to set up mirror maker to mirror data from EU dc to US dc. The > network delay is ~150 ms. In recent test; we realized that mirror maker is >

Re: Consumer offsets partitions size much bigger than others

errors in the event you don't have enough brokers. -James > On Jul 18, 2017, at 8:44 AM, Luciano Afranllie > wrote: > > Hi > > One of our Kafka brokers was running out of disk space and when we checked > the file size in the kafka log dir we observed the following

Re: Mirroring multiple clusters into one

Answers inline below. -James Sent from my iPhone > On Jul 7, 2017, at 1:18 AM, Vahid S Hashemian > wrote: > > James, > > Thanks for sharing your thoughts and experience. > Could you please also confirm whether > - you do any encryption for the mirrored data? Not at t

Re: Mirroring multiple clusters into one

about what to look at. https://www.slideshare.net/mobile/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600 -James Sent from my iPhone > On Jul 6, 2017, at 1:00 AM, Vahid S Hashemian > wrote: > > The literature suggests running the MM on the target cluster when possible > (with

Re: mirroring Kafka while preserving the order

. MirrorMaker supports custom message handlers. You can use those to map the source partition to the destination partition, which will allow you to avoid the above two problems. Here's an example of how to do it. https://github.com/gwenshap/kafka-examples/tree/master/MirrorMakerHandler -James Sent

question about document

Hello, At https://kafka.apache.org/intro, I found a sentence: Each partition has one server which acts as the "leader" and zero or more servers which act as "followers". In my opinion, it is possible that more than one brokers exist on a machine with different ports and different bro

Re: Slow Consumer Group Startup

Bryan, This sounds related to https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance and https://issues.apache.org/jira/browse/KAFKA-4925. -James > On Jun 13, 2017, at 7:02 AM, Bryan Baugher wrote: > > The topics already exist prior to

Re: [ANNOUNCE] New committer: Damian Guy

Congrats Damian! -James > On Jun 9, 2017, at 1:34 PM, Guozhang Wang wrote: > > Hello all, > > > The PMC of Apache Kafka is pleased to announce that we have invited Damian > Guy as a committer to the project. > > Damian has made tremendous contribution

Re: topics stuck in "Leader: -1" after crash while migrating topics

For what it's worth, shutting down the entire cluster and then restarting it did address this issue. I'd love anyone's thoughts on what the "correct" fix would be here. On Fri, Apr 28, 2017 at 10:58 AM, James Brown wrote: > The following is also appearing in the l

Re: topics stuck in "Leader: -1" after crash while migrating topics

The following is also appearing in the logs a lot, if anyone has any ideas: INFO Partition [easypost.syslog,7] on broker 1: Cached zkVersion [647] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition) On Fri, Apr 28, 2017 at 10:43 AM, James Brown wrote: > We're

topics stuck in "Leader: -1" after crash while migrating topics

to only have whatever the third broker was in their replica-set as their replica set? Do I need to temporarily enable unclean elections? I've never seen a cluster fail this way...​ -- James Brown Engineer

Re: stunning error - Request of length 1550939497 is not valid, it is larger than the maximum size of 104857600 bytes

e some protocol that uses the word "\quit"? Like IRC or SMTP or IMAP something? I'm not sure. Anyway, thought you might find that interesting! -James > On Dec 12, 2016, at 9:39 AM, Todd Palino wrote: > > Are you actually getting requests that are 1.3 GB in size, or

how can I contribute to this project?

Hi Because I love this project, so I want to take part of it. But I'm brand new to opensource project. How can I get started to make contribution? Can you give me some advise or something? By the way, I already have JIRA account which called "james.c" Sincerely, James.C

Re: ISR churn

aven't been able it isolate a cause. I think it's related to flaky connections to the zookeeper cluster. -James > On Mar 22, 2017, at 11:46 AM, Marcos Juarez wrote: > > We're seeing the same exact pattern of ISR shrinking/resizing, mostly on > partitions with the lar

Re: Offset commit request failing

ssage gets too big to go through the topic. There is a long thread here that discusses it. I don't remember what specific actions came out of that discussion. http://search-hadoop.com/m/Kafka/uyzND1yd26N1rFtRd1?subj=+DISCUSS+scalability+limits+in+the+coordinator -James Sent from my iPhone

  1   2   3   >