Re: Public IPs for brokers

Hi Himanshu, Short answer: yes. The way a Kafka client works is to connect to one of the given bootstrap brokers and ask it about the rest of the cluster. The client then connects to those brokers as necessary. So in general a client will need to connect to all brokers in the cluster and therefor

Public IPs for brokers

Hi all, do I have to necessarily have public IPs for all the brokers? Since we can give a few of the IPs in bootstrap.servers config.

ingesting web server logs, or log4j log files from a boss server

Hi all. Please advise, a real noob here still, unpacking how the stack still works... if I have a mySQL server, or a web server, or a 2 node JBOSS cluster. If I want to use the mysql connector to connect to the MySQL DB to pull data using CDC... then I need to install the Kafka stack on the DB s

When does kafka connect commit messages to kafka broker ?

Hello. I would like to ask question regarding to kafka connect. I am trying to import over 1 million+ records from postgres to kafka broker using kafka connect using jdbc driver . However, it seems that topic for storing messages is not created immediately. I have built my kafka-connect and brok

Re: MirrorMaker 2

Thanks On Tue, Jan 14, 2020 at 8:47 AM Ryanne Dolan wrote: > Take a look at the DefaultConfigPropertyFilter class, which supports > customizable blacklists via config.properties.blacklist. > > Ryanne > > On Tue, Jan 14, 2020, 6:05 AM Vishal Santoshi > wrote: > > > Thank you for the prompt reply

Re: Request for joining

If you want to join the mailing list(s), it's self-service. Instructions on how to subscribe are on the webpage: https://kafka.apache.org/contact -Matthias On 1/13/20 9:55 PM, Sunil CHAUDHARI wrote: > Hi, > Please allow me to join kafka discussion. > > > Thanks & regards, > Sunil. > CONFIDENT

Kafka send() call, ack, and message writes to the filesystem

I have a question related to the semantics of a producer send and the get calls on the future returned by the send call. - It is a Java application, using the Kafka Java client library - The application is set up to use 3 replicas and using acks=all for the producer - the application is using a

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Congratulations Colin, Vahid, Manikumar! Guozhang On Tue, Jan 14, 2020 at 10:53 AM Matthias J. Sax wrote: > Congrats! > > On 1/14/20 10:49 AM, Rajini Sivaram wrote: > > Congratulations Colin, Vahid and Manikumar! > > > > Regards, > > Rajini > > > > On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Congrats! On 1/14/20 10:49 AM, Rajini Sivaram wrote: > Congratulations Colin, Vahid and Manikumar! > > Regards, > Rajini > > On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison > wrote: > >> Congrats Colin, Vahid and Manikumar! >> >> On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma wrote: >>> >>> Congr

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Congrats Colin, Vahid and Manikumar! On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma wrote: > > Congratulations Colin, Vahid and Manikumar! > > Ismael > > On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira wrote: > > > Hi everyone, > > > > I'm happy to announce that Colin McCabe, Vahid Hashemian and Manik

Re: Mirror Maker 2 problems

Nils, a REST API for MM2 was discussed at length, but so far the community hasn't settled on a plan. For now, if you need to monitor MM2's Connectors over HTTP, you can use a bunch of Connect clusters and manually configure them to run MirrorSourceConnector etc. This has some advantages, e.g. you c

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Congratulations Colin, Vahid and Manikumar! Ismael On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira wrote: > Hi everyone, > > I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar > Reddy are now members of Apache Kafka PMC. > > Colin and Manikumar became committers on Sept 2018 an

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Congrats, Colin, Vahid, and Manikumar! A great accomplishment, reflecting your great work. -John On Tue, Jan 14, 2020, at 11:33, Bill Bejeck wrote: > Congrats Colin, Vahid and Manikumar! Well deserved. > > -Bill > > On Tue, Jan 14, 2020 at 12:30 PM Gwen Shapira wrote: > > > Hi everyone, > >

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Congrats Colin, Vahid and Manikumar! Well deserved. -Bill On Tue, Jan 14, 2020 at 12:30 PM Gwen Shapira wrote: > Hi everyone, > > I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar > Reddy are now members of Apache Kafka PMC. > > Colin and Manikumar became committers on Sept

[ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Hi everyone, I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar Reddy are now members of Apache Kafka PMC. Colin and Manikumar became committers on Sept 2018 and Vahid on Jan 2019. They all contributed many patches, code reviews and participated in many KIP discussions. We app

Re: Request for joining

Hi, Please allow me to join kafka discussion. Thanks and Regards, Himanshu Shukla On Tue, Jan 14, 2020 at 10:32 PM Sunil CHAUDHARI wrote: > Hi, > Please allow me to join kafka discussion. > > > Thanks & regards, > Sunil. > CONFIDENTIAL NOTE: > The information contained in this email is inten

Re: Streams, Kafka windows

You can try to convert the final resultant stream to table. Check this page for more info: https://docs.confluent.io/current/streams/faq.html#how-can-i-convert-a-kstream-to-a-ktable-without-an-aggregation-step This way table would always contain the latest (single) record for a given key. Sachin

Request for joining

Hi, Please allow me to join kafka discussion. Thanks & regards, Sunil. CONFIDENTIAL NOTE: The information contained in this email is intended only for the use of the individual or entity named above and may contain information that is privileged, confidential and exempt from disclosure under ap

Streams, Kafka windows

Hi, My name is Viktor. I'm currently working with Kafka streams and have several questions about Kafka and I can not find answers in the official docs. 1. Why suppress functionality does not work with Hopping windows? How to make it work? Example of the code: KStream finalStream = source

Re: Mirror Maker 2 problems

Another question is there anyway to open up a rest port like standard 8083 when running ./bin/connect-mirror-maker.sh config/connect-mirror-maker.properties have tried adding rest.port=8083 but does not seem to work. Would like to be able to query the status of the connector via rest. On Tue, J

Re: Mirror Maker 2 problems

Hello, yes that seems todo the trick. Another questions is there any documentation somewhere describing how the offset translations topics MM2 creates actually works? We need to write code ourself until this PR is merged I guess. https://github.com/apache/kafka/pull/7577 Thanks for the help. On

Re: MirrorMaker 2

Take a look at the DefaultConfigPropertyFilter class, which supports customizable blacklists via config.properties.blacklist. Ryanne On Tue, Jan 14, 2020, 6:05 AM Vishal Santoshi wrote: > Thank you for the prompt reply, Very much appreciated. I am not sure > disable config sync is an option fo

upper bound of the number of records

Hi, I am running a 3 broker Kafka cluster. While doing the performance test, I am seeing that the broker is not processing the records more than 50MB/sec. if I am running multiple producer threads on the same machine, even this 50K is getting divided into 50K/no_of_prod_thread. is there any co

Re: MirrorMaker 2

Thank you for the prompt reply, Very much appreciated. I am not sure disable config sync is an option for us. How do I blacklist retention.ms ? On Tue, Jan 14, 2020 at 12:47 AM Ryanne Dolan wrote: > Vishal, there is no support for overriding topic configuration like > retention. Instead, remot

Re: Mirror Maker 2 problems

Hi Nils Can you try with the latest mm2 default config file found at https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties. Please feel free to reach out again if you are still stuck. Thanks Karan On Tue, Jan 14, 2020 at 1:48 PM Nils Hermansson < nils.hermans...@get

Mirror Maker 2 problems

Hello, I am trying to setup a replication between 2 clusters with MM2. The goal is to be able to disksnapshot the secondary cluster for backups hence we only want one broker on the secondary cluster. It's very important for us that we can replicate the offsets which MM2 should solve for us. Cluste