[DISCUSS] KAFKA-4169: KafkaProducer max.request.size message wrong when using compression
Hi, my name is Pere, Software Engineer turned Consultant for the last few years, :-/ , but keep doing Kafka quite a lot of my day-to-day. I know not a sexy issue, but last few days, I worked with customers who again hit https://issues.apache.org/jira/browse/KAFKA-4169 when using producing with compression, getting a few uncomfortable RecordTooLargeExceptions. While I see and agree with the community, this issue is not that important, it keeps annoying starters, so I thought to try giving it a try to do something here. I updated the issue with some comments of mine, but in a nutshell, unless I'm missing something, it certainly could be. I think the max.request.size config and the related check are wrong and problematic, Jorge already mentioned that in the ticket. In my opinion, it mixes requests (batches) with single messages, and it does not take into account any compression. I can understand when people get confused, and so do I from time to time. Before starting a KIP, and writing a ton I wanted to drop here a message and see how you feel about possible solutions I have thought. To my understanding, all might need a KIP, but please let me know if I got it wrong. * As Jorge proposed in the ticket, use the CompressionRateEstimator to complement the existing check and be more accurate for single messages than it is right now. All with having in mind that it will be working with an estimator. This will require, in my mind to add a new configuration for a single message size, isn't it? * Remove completely the fail-fast check earlier mentioned and is currently in place. Introduce a new check at the tryAppend methods within the RecordAccumulator. The check-in this case will be using, as in option 1, the CompressionRateEstimator. In this case, I see the current variable as being more accurate, as we will be doing request level (ProducerBatch) level checks. What do you think? Looking forward to help out, -- Pere Urbon-Bayes Software Architect https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [ANNOUNCE] New committer: Stanislav Kozlovski
Congrats Stan, very well deserved. -- Pere On Wed, Jan 18, 2023 at 7:51 AM Yash Mayya wrote: > Congratulations, Stan! > > On Tue, Jan 17, 2023 at 9:20 PM Jun Rao wrote: > > > Hi, Everyone, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer > > Stanislav Kozlovski. > > > > Stan has been contributing to Apache Kafka since June 2018. He made > various > > contributions including the following KIPs. > > > > KIP-455: Create an Administrative API for Replica Reassignment > > KIP-412: Extend Admin API to support dynamic application log levels > > > > Congratulations, Stan! > > > > Thanks, > > > > Jun (on behalf of the Apache Kafka PMC) > > > -- Pere Urbon-Bayes Software Architect https://twitter.com/purbon https://www.linkedin.com/in/purbon/
[DISCUSS] KIP-703: Add a metrics for reporting idle connections closed
Hi, I would love to kick start a discussion for KIP-703, the proposal is available https://cwiki.apache.org/confluence/display/KAFKA/KIP-703%3A+Add+a+metrics+for+reporting+idle+connections+closed It is a minor change but certainly helps to improve the ops day2day life. What do you think? Looking forward to hearing your thoughts and feedback. -- Pere Urbon-Bayes Software Architect https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
True, the number is duplicated. do you know how can we get the problem fix? I was not aware, I missed, sorry the need to add the KIP to the table. -- Pere Missatge de Jorge Esteban Quilcate Otoya del dia dt., 3 de set. 2019 a les 13:43: > Hi Pere, > > Have you add your KIP to the list here > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals? > > I found the KIP number assigned to another. > > > > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes > wrote: > >> Thanks for your time Harsha, >>anyone else with comments? looking forward to hearing from you. >> >> Stupid question: when do you move from discussion to vote? >> >> Missatge de Harsha Chintalapani del dia dv., 30 d’ag. >> 2019 a les 21:59: >> >> > Thanks Pere. KIP looks good to me. >> > -Harsha >> > >> > >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes < >> pere.ur...@gmail.com> >> > wrote: >> > >> >> Not really, >> >> my idea is to keep the JAAS parameter, so people don't see major >> >> changes. But if you pass a properties file, then this takes precedence >> over >> >> the other, with the idea that you can do sasl as well with the >> properties >> >> files. >> >> >> >> Makes sense? >> >> >> >> -- Pere >> >> >> >> Missatge de Harsha Chintalapani del dia dv., 30 >> d’ag. >> >> 2019 a les 19:00: >> >> >> >>> Hi Pere, >> >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka >> makes >> >>> sense. >> >>> "The changes are planned to be introduced in a compatible way, by >> >>> keeping the current JAAS variable precedence." >> >>> Can you elaborate a bit here. If the user configures a JAAS file with >> >>> Client section it will take precedence over zookeeper SSL configs? >> >>> >> >>> Thanks, >> >>> Harsha >> >>> >> >>> >> >>> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes < >> pere.ur...@gmail.com> >> >>> wrote: >> >>> >> >>>> Hi, >> >>>> quick question, I saw in another mail that 2.4 release is planned for >> >>>> September. I think it would be really awesome to have this for this >> >>>> release, do you think we can make it? >> >>>> >> >>>> -- Pere >> >>>> >> >>>> Missatge de Pere Urbón Bayes del dia dj., 29 >> >>>> d’ag. 2019 a les 20:10: >> >>>> >> >>>> Hi, >> >>>> this is my first KIP for a change in Apache Kafka, so I'm really need >> >>>> to the process. Looking forward to hearing from you and learn the >> best >> >>>> ropes here. >> >>>> >> >>>> I would like to propose this KIP-515 to enable the ZookeeperClients >> to >> >>>> take full advantage of the TLS communication in the new Zookeeper >> 3.5.5. >> >>>> Specially interesting it the Zookeeper Security Migration, that >> without >> >>>> this change will not work with TLS, disabling users to use ACLs when >> the >> >>>> Zookeeper cluster use TLS. >> >>>> >> >>>> link: >> >>>> >> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication >> >>>> >> >>>> Looking forward to hearing from you on this, >> >>>> >> >>>> /cheers >> >>>> >> >>>> -- >> >>>> Pere Urbon-Bayes >> >>>> Software Architect >> >>>> http://www.purbon.com >> >>>> https://twitter.com/purbon >> >>>> https://www.linkedin.com/in/purbon/ >> >>>> >> >>>> -- >> >>>> Pere Urbon-Bayes >> >>>> Software Architect >> >>>> http://www.purbon.com >> >>>> https://twitter.com/purbon >> >>>> https://www.linkedin.com/in/purbon/ >> >>>> >> >>> >> >>> >> >> >> >> -- >> >> Pere Urbon-Bayes >> >> Software Architect >> >> http://www.purbon.com >> >> https://twitter.com/purbon >> >> https://www.linkedin.com/in/purbon/ >> >> >> > >> > >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi Ismael, fair point and totally reasonable point. I added the CLI part to bring compatibility, but I can agree with not doing it right now per compatibility reasons. However, I do have a question for you: You say most of the CLI tools, is there anyone in your mind that you recall that would break if the cluster uses a TLS activated Zookeeper cluster? Only one I can probably think right now is the Zookeeper Security Migration tool, right? If I clean the others, would you agree to move the KIP forward? -- Pere Missatge de Ismael Juma del dia dt., 3 de set. 2019 a les 14:53: > Hi Pere, > > Thanks for the KIP. With regards to the CLI tools, most of them support > direct access to ZK for compatibility reasons and we encourage usage of the > Kafka protocol instead. I am not sure we should be extending them as > described in the KIP. What are your thoughts on that? > > Ismael > > On Thu, Aug 29, 2019 at 11:11 AM Pere Urbón Bayes > wrote: > >> Hi, >> this is my first KIP for a change in Apache Kafka, so I'm really need to >> the process. Looking forward to hearing from you and learn the best ropes >> here. >> >> I would like to propose this KIP-515 to enable the ZookeeperClients to >> take >> full advantage of the TLS communication in the new Zookeeper 3.5.5. >> Specially interesting it the Zookeeper Security Migration, that without >> this change will not work with TLS, disabling users to use ACLs when the >> Zookeeper cluster use TLS. >> >> link: >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication >> >> Looking forward to hearing from you on this, >> >> /cheers >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Thanks for your time Harsha, anyone else with comments? looking forward to hearing from you. Stupid question: when do you move from discussion to vote? Missatge de Harsha Chintalapani del dia dv., 30 d’ag. 2019 a les 21:59: > Thanks Pere. KIP looks good to me. > -Harsha > > > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes > wrote: > >> Not really, >> my idea is to keep the JAAS parameter, so people don't see major >> changes. But if you pass a properties file, then this takes precedence over >> the other, with the idea that you can do sasl as well with the properties >> files. >> >> Makes sense? >> >> -- Pere >> >> Missatge de Harsha Chintalapani del dia dv., 30 d’ag. >> 2019 a les 19:00: >> >>> Hi Pere, >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka makes >>> sense. >>> "The changes are planned to be introduced in a compatible way, by >>> keeping the current JAAS variable precedence." >>> Can you elaborate a bit here. If the user configures a JAAS file with >>> Client section it will take precedence over zookeeper SSL configs? >>> >>> Thanks, >>> Harsha >>> >>> >>> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes >>> wrote: >>> >>>> Hi, >>>> quick question, I saw in another mail that 2.4 release is planned for >>>> September. I think it would be really awesome to have this for this >>>> release, do you think we can make it? >>>> >>>> -- Pere >>>> >>>> Missatge de Pere Urbón Bayes del dia dj., 29 >>>> d’ag. 2019 a les 20:10: >>>> >>>> Hi, >>>> this is my first KIP for a change in Apache Kafka, so I'm really need >>>> to the process. Looking forward to hearing from you and learn the best >>>> ropes here. >>>> >>>> I would like to propose this KIP-515 to enable the ZookeeperClients to >>>> take full advantage of the TLS communication in the new Zookeeper 3.5.5. >>>> Specially interesting it the Zookeeper Security Migration, that without >>>> this change will not work with TLS, disabling users to use ACLs when the >>>> Zookeeper cluster use TLS. >>>> >>>> link: >>>> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication >>>> >>>> Looking forward to hearing from you on this, >>>> >>>> /cheers >>>> >>>> -- >>>> Pere Urbon-Bayes >>>> Software Architect >>>> http://www.purbon.com >>>> https://twitter.com/purbon >>>> https://www.linkedin.com/in/purbon/ >>>> >>>> -- >>>> Pere Urbon-Bayes >>>> Software Architect >>>> http://www.purbon.com >>>> https://twitter.com/purbon >>>> https://www.linkedin.com/in/purbon/ >>>> >>> >>> >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> > > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Not really, my idea is to keep the JAAS parameter, so people don't see major changes. But if you pass a properties file, then this takes precedence over the other, with the idea that you can do sasl as well with the properties files. Makes sense? -- Pere Missatge de Harsha Chintalapani del dia dv., 30 d’ag. 2019 a les 19:00: > Hi Pere, > Thanks for the KIP. Enabling SSL for zookeeper for Kafka makes > sense. > "The changes are planned to be introduced in a compatible way, by keeping > the current JAAS variable precedence." > Can you elaborate a bit here. If the user configures a JAAS file with > Client section it will take precedence over zookeeper SSL configs? > > Thanks, > Harsha > > > > On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes > wrote: > >> Hi, >> quick question, I saw in another mail that 2.4 release is planned for >> September. I think it would be really awesome to have this for this >> release, do you think we can make it? >> >> -- Pere >> >> Missatge de Pere Urbón Bayes del dia dj., 29 >> d’ag. 2019 a les 20:10: >> >> Hi, >> this is my first KIP for a change in Apache Kafka, so I'm really need to >> the process. Looking forward to hearing from you and learn the best ropes >> here. >> >> I would like to propose this KIP-515 to enable the ZookeeperClients to >> take full advantage of the TLS communication in the new Zookeeper 3.5.5. >> Specially interesting it the Zookeeper Security Migration, that without >> this change will not work with TLS, disabling users to use ACLs when the >> Zookeeper cluster use TLS. >> >> link: >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication >> >> Looking forward to hearing from you on this, >> >> /cheers >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> > > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
It helps as a beginner! thanks for sharing! any tip for getting reviewers to take a look? :pray:, :pray: ;-) -- Pere Missatge de Colin McCabe del dia dv., 30 d’ag. 2019 a les 18:57: > Hi Pere, > > It will depend on how quickly the KIP can be voted in, and then > implemented. > > Manikumar Reddy is going to be the release manager for Apache Kafka > 2.4.0. Right now, the KIP freeze is scheduled for Sep 25, 2019. A KIP > must be accepted by this date in order to be part of 2.4.0. (Being > accepted doesn't mean it will be part of 2.4.0, though-- it also has to be > implemented.) Hope this helps. > > cheers, > Colin > > > On Fri, Aug 30, 2019, at 07:50, Pere Urbón Bayes wrote: > > Hi, > > quick question, I saw in another mail that 2.4 release is planned for > > September. I think it would be really awesome to have this for this > > release, do you think we can make it? > > > > -- Pere > > > > Missatge de Pere Urbón Bayes del dia dj., 29 > d’ag. > > 2019 a les 20:10: > > > > > Hi, > > > this is my first KIP for a change in Apache Kafka, so I'm really need > to > > > the process. Looking forward to hearing from you and learn the best > ropes > > > here. > > > > > > I would like to propose this KIP-515 to enable the ZookeeperClients to > > > take full advantage of the TLS communication in the new Zookeeper > 3.5.5. > > > Specially interesting it the Zookeeper Security Migration, that without > > > this change will not work with TLS, disabling users to use ACLs when > the > > > Zookeeper cluster use TLS. > > > > > > link: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > > > > > Looking forward to hearing from you on this, > > > > > > /cheers > > > > > > -- > > > Pere Urbon-Bayes > > > Software Architect > > > http://www.purbon.com > > > https://twitter.com/purbon > > > https://www.linkedin.com/in/purbon/ > > > > > > > > > -- > > Pere Urbon-Bayes > > Software Architect > > http://www.purbon.com > > https://twitter.com/purbon > > https://www.linkedin.com/in/purbon/ > > > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi, quick question, I saw in another mail that 2.4 release is planned for September. I think it would be really awesome to have this for this release, do you think we can make it? -- Pere Missatge de Pere Urbón Bayes del dia dj., 29 d’ag. 2019 a les 20:10: > Hi, > this is my first KIP for a change in Apache Kafka, so I'm really need to > the process. Looking forward to hearing from you and learn the best ropes > here. > > I would like to propose this KIP-515 to enable the ZookeeperClients to > take full advantage of the TLS communication in the new Zookeeper 3.5.5. > Specially interesting it the Zookeeper Security Migration, that without > this change will not work with TLS, disabling users to use ACLs when the > Zookeeper cluster use TLS. > > link: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > Looking forward to hearing from you on this, > > /cheers > > -- > Pere Urbon-Bayes > Software Architect > http://www.purbon.com > https://twitter.com/purbon > https://www.linkedin.com/in/purbon/ > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
[DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi, this is my first KIP for a change in Apache Kafka, so I'm really need to the process. Looking forward to hearing from you and learn the best ropes here. I would like to propose this KIP-515 to enable the ZookeeperClients to take full advantage of the TLS communication in the new Zookeeper 3.5.5. Specially interesting it the Zookeeper Security Migration, that without this change will not work with TLS, disabling users to use ACLs when the Zookeeper cluster use TLS. link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication Looking forward to hearing from you on this, /cheers -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: Permissions to create a KIP for KAFKA-8843
Working on it, looking forward to do my best with my first KIP, I will send details for review as soon as I have them. -- Pere Missatge de Colin McCabe del dia dj., 29 d’ag. 2019 a les 19:06: > On Thu, Aug 29, 2019, at 09:27, Pere Urbón Bayes wrote: > > Thanks, > > yes I know the KIP-500, how realistic is to have it landing for 2.4? if > > not really, end of the day we will need something like KAFKA-8843 for > this > > version. > > > > Looking forward to help out. > > > > Hi Pere, > > Thanks for contributing. KIP-500 will not be implemented in 2.4. > > KAFKA-8843 seems like a good improvement. Since it involves changing > command-line arguments, I think a KIP will be needed. However, it should > be a relatively short one if the change is straightforward. > > best, > Colin > > > > -- Pere > > > > Missatge de Guozhang Wang del dia dj., 29 d’ag. > 2019 a > > les 17:35: > > > > > Hello Pere, > > > > > > Thanks for you interest in contributing to Kafka, I've added you to the > > > contributors list and you should be able to create wiki pages now. > > > > > > BTW there's an on-going KIP-500 which aims at removing ZK dependency of > > > Kafka recently; depending when it would be voted and be adopted, I > think > > > it's still worth doing KAFKA-8843. > > > > > > Guozhang > > > > > > > > > On Thu, Aug 29, 2019 at 3:56 AM Pere Urbón Bayes > > > > wrote: > > > > > >> Hi, > > >> I would like to create, and start the process to fix KAFKA-8843, so > as I > > >> understand it I should first create a KIP. > > >> > > >> I registered to the wiki: > > >> email: pere.ur...@gmail.com > > >> user: pere.urbon > > >> > > >> would you be so nice to provide permissions to create the KIP? > > >> > > >> thanks a lot, > > >> > > >> -- > > >> Pere Urbon-Bayes > > >> Software Architect > > >> http://www.purbon.com > > >> https://twitter.com/purbon > > >> https://www.linkedin.com/in/purbon/ > > >> > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > -- > > Pere Urbon-Bayes > > Software Architect > > http://www.purbon.com > > https://twitter.com/purbon > > https://www.linkedin.com/in/purbon/ > > > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: Permissions to create a KIP for KAFKA-8843
Thanks, yes I know the KIP-500, how realistic is to have it landing for 2.4? if not really, end of the day we will need something like KAFKA-8843 for this version. Looking forward to help out. -- Pere Missatge de Guozhang Wang del dia dj., 29 d’ag. 2019 a les 17:35: > Hello Pere, > > Thanks for you interest in contributing to Kafka, I've added you to the > contributors list and you should be able to create wiki pages now. > > BTW there's an on-going KIP-500 which aims at removing ZK dependency of > Kafka recently; depending when it would be voted and be adopted, I think > it's still worth doing KAFKA-8843. > > Guozhang > > > On Thu, Aug 29, 2019 at 3:56 AM Pere Urbón Bayes > wrote: > >> Hi, >> I would like to create, and start the process to fix KAFKA-8843, so as I >> understand it I should first create a KIP. >> >> I registered to the wiki: >> email: pere.ur...@gmail.com >> user: pere.urbon >> >> would you be so nice to provide permissions to create the KIP? >> >> thanks a lot, >> >> -- >> Pere Urbon-Bayes >> Software Architect >> http://www.purbon.com >> https://twitter.com/purbon >> https://www.linkedin.com/in/purbon/ >> > > > -- > -- Guozhang > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Permissions to create a KIP for KAFKA-8843
Hi, I would like to create, and start the process to fix KAFKA-8843, so as I understand it I should first create a KIP. I registered to the wiki: email: pere.ur...@gmail.com user: pere.urbon would you be so nice to provide permissions to create the KIP? thanks a lot, -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
[Discuss] Question on KIP-298: Error Handling in Kafka Connect
Hi, a quick question on the KIP-298 Dead letter queue, as I read from the KIP is only available for the Sink connectors. While I know the challenges of defining a dead-letter queue for the incoming messages, I wanted ask/discuss what is the sense in here for this, do you completely discard the option? I sort of see it useful for messages that where pulled from the source, but somehow could not be ingested in Kafka, might be because of serialisation for example. What do you think? -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
Re: Kafka tests on a remote cluster
Hi, if I understand your question properly you are aiming to validate failure scenarios. I usually see this done for learning purposes, basically to answer how Kafka would react to such and such situation? or to validate the current setup / configuration an Organisation have. For the first one, or as well the second one but not directly on the cluster, colleagues of mine actually wrote https://github.com/Dabz/kafka-boom-boom, this uses docker to simulate the scenarios. I've actually seen this approach used often. This days I would only run a chaos testing if supported by a tooling like the Netflix Chaos Monkey (https://github.com/Netflix/chaosmonkey) or the other tooling available for the cloud providers, Kubernetes or Cloud Foundry. A nice list is available from https://github.com/dastergon/awesome-chaos-engineering#notable-tools Cheers -- Pere Missatge de Parviz deyhim del dia dj., 27 de des. 2018 a les 21:57: > +dev@kafka.apache.org > > On Wed, Dec 26, 2018 at 8:53 PM Parviz deyhim wrote: > > > Thanks fair points. Probably best if I simplify the question: How does > > Kafka community run tests besides using mocked local Kafka components? > > Surely there are tests to confirm different failure scenarios such as > > losing a broker in a real clustered environment (multi node cluster with > > Ip, port, hostnsmes and etc). The answer would be a good starting point > for > > me. > > > > On Wed, Dec 26, 2018 at 6:11 PM Stephen Powis > > wrote: > > > >> Without looking into how the integration tests work my best guess is > >> within > >> the context they were written to run in, it doesn't make sense to run > them > >> against a remote cluster. The "internal" cluster is running the same > >> code, > >> so why require having to coordinate with an external dependency? > >> > >> For the use case you gave, and I'm not sure if tests exist that cover > this > >> behavior or not -- Running the brokers locally in the context of the > tests > >> mean that those tests have control over the brokers (IE shut them off, > >> restart them, etc.. programmatically) and validate behavior. To > >> coordinate > >> these operations on a remote broker would be significantly more > difficult. > >> > >> Not sure this helps...but perhaps you're either asking the wrong > questions > >> or trying to go about solving your problem using the wrong set of tools? > >> My gut feeling says if you want to do a full scale multi-server load / > HA > >> test, Kafka's test suite is not the best place to start. > >> > >> Stephen > >> > >> > >> > >> On Thu, Dec 27, 2018 at 10:53 AM Parviz deyhim > wrote: > >> > >> > Hi, > >> > > >> > I'm looking to see who has done this before and get some guidance. On > >> > frequent basis I like to run basic tests on a remote Kafka cluster > while > >> > some random chaos/faults are being performed. In other words I like to > >> run > >> > chaos engineering tasks (network outage, disk outage, etc) and see how > >> > Kafka behaves. For example: > >> > > >> > 1) bring some random Broker node down > >> > 2) send 2000 messages > >> > 3) consumes messages > >> > 4) confirm there's no data loss > >> > > >> > My questions: I'm pretty sure most of the scenarios I'm looking to > test > >> > have been covered under Kafka's integration,unit and other existing > >> tests. > >> > What I cannot figure out is how to run those tests on a remote cluster > >> vs. > >> > a local one which the tests seems to run on. For example I like to run > >> the > >> > following command but the tests to be executed on a remote cluster of > my > >> > choice: > >> > > >> > ./gradlew cleanTest integrationTest > >> > > >> > Any guidance/help would be appreciated. > >> > > >> > Thanks > >> > > >> > > > -- Pere Urbon-Bayes Software Architect http://www.purbon.com https://twitter.com/purbon https://www.linkedin.com/in/purbon/
