Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Hi, I updated the KIP and added an `OutputVerifier` class. Looking forward to your feedback. I will start the Vote thread in parallel. -Matthias On 1/17/18 11:40 AM, Matthias J. Sax wrote: > Saïd, > > sorry for late reply. Avro records are basically supported. However, > there are no plans atm to integrate the test driver with a schema > registry. Note, that Apache Kafka does not provide a schema registry by > itself. If we integrate one RS, we need to integrate all of them -- that > is something Apache Kafka should not do from my point of view. > > Thus, if you want to use ArvoSerdes that use a schema registry, those > should mock be mocked accordingly. However, there is nothing this KIP > can help with this regard IMHO. > > I left a comment on the ticket to pointed out. :) > > > -Matthias > > On 1/17/18 1:29 AM, Saïd Bouras wrote: >> Matthias, >> >> What about testing topology that use avro schema ? Have you read my >> previous response ? >> >> Thanks. >> >> On Wed, Jan 17, 2018 at 3:34 AM Matthias J. Sax>> wrote: >> >>> Colin, >>> >>> the TopologyTestDriver does not connect to any broker and simulates >>> processing of single-partitioned input topics purely in-memory (the >>> driver is basically a mock for a StreamThread). This is sufficient to >>> test basic business logic. For more complex topologies that are actually >>> divided into sub-topologies and connected via topics, the driver detects >>> this case and does an in-memory forward. >>> >>> >>> -Matthias >>> >>> On 1/16/18 10:08 AM, Colin McCabe wrote: Thanks, Matthias, this looks great. It seems like these APIs could either be used against mock objects, or >>> against real brokers running in the same process. Is there a way for the >>> user to select which they want when using the API? Sorry if it's in the >>> KIP and I missed it. cheers, Colin On Thu, Jan 11, 2018, at 18:06, Matthias J. Sax wrote: > Dear Kafka community, > > I want to propose KIP-247 to add public test utils to the Streams API. > The goal is to simplify testing of Kafka Streams applications. > > Please find details in the wiki: > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams > > This is an initial KIP, and we hope to add more utility functions later. > Thus, this KIP is not comprehensive but a first step. Of course, we can > enrich this initial KIP if we think it falls too short. But we should > not aim to be comprehensive to keep the scope manageable. > > In fact, I think we should add some more helpers to simplify result > verification. I will update the KIP with this asap. Just wanted to start > the discussion early on. > > An initial WIP PR can be found here: > https://github.com/apache/kafka/pull/4402 > > I also included the user-list (please hit "reply-all" to include both > lists in this KIP discussion). > > Thanks a lot. > > > -Matthias > > > Email had 1 attachment: > + signature.asc > 1k (application/pgp-signature) >>> >>> >> > signature.asc Description: OpenPGP digital signature
Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Saïd, sorry for late reply. Avro records are basically supported. However, there are no plans atm to integrate the test driver with a schema registry. Note, that Apache Kafka does not provide a schema registry by itself. If we integrate one RS, we need to integrate all of them -- that is something Apache Kafka should not do from my point of view. Thus, if you want to use ArvoSerdes that use a schema registry, those should mock be mocked accordingly. However, there is nothing this KIP can help with this regard IMHO. I left a comment on the ticket to pointed out. :) -Matthias On 1/17/18 1:29 AM, Saïd Bouras wrote: > Matthias, > > What about testing topology that use avro schema ? Have you read my > previous response ? > > Thanks. > > On Wed, Jan 17, 2018 at 3:34 AM Matthias J. Sax> wrote: > >> Colin, >> >> the TopologyTestDriver does not connect to any broker and simulates >> processing of single-partitioned input topics purely in-memory (the >> driver is basically a mock for a StreamThread). This is sufficient to >> test basic business logic. For more complex topologies that are actually >> divided into sub-topologies and connected via topics, the driver detects >> this case and does an in-memory forward. >> >> >> -Matthias >> >> On 1/16/18 10:08 AM, Colin McCabe wrote: >>> Thanks, Matthias, this looks great. >>> >>> It seems like these APIs could either be used against mock objects, or >> against real brokers running in the same process. Is there a way for the >> user to select which they want when using the API? Sorry if it's in the >> KIP and I missed it. >>> >>> cheers, >>> Colin >>> >>> >>> On Thu, Jan 11, 2018, at 18:06, Matthias J. Sax wrote: Dear Kafka community, I want to propose KIP-247 to add public test utils to the Streams API. The goal is to simplify testing of Kafka Streams applications. Please find details in the wiki: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams This is an initial KIP, and we hope to add more utility functions later. Thus, this KIP is not comprehensive but a first step. Of course, we can enrich this initial KIP if we think it falls too short. But we should not aim to be comprehensive to keep the scope manageable. In fact, I think we should add some more helpers to simplify result verification. I will update the KIP with this asap. Just wanted to start the discussion early on. An initial WIP PR can be found here: https://github.com/apache/kafka/pull/4402 I also included the user-list (please hit "reply-all" to include both lists in this KIP discussion). Thanks a lot. -Matthias Email had 1 attachment: + signature.asc 1k (application/pgp-signature) >> >> > signature.asc Description: OpenPGP digital signature
Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Matthias, What about testing topology that use avro schema ? Have you read my previous response ? Thanks. On Wed, Jan 17, 2018 at 3:34 AM Matthias J. Saxwrote: > Colin, > > the TopologyTestDriver does not connect to any broker and simulates > processing of single-partitioned input topics purely in-memory (the > driver is basically a mock for a StreamThread). This is sufficient to > test basic business logic. For more complex topologies that are actually > divided into sub-topologies and connected via topics, the driver detects > this case and does an in-memory forward. > > > -Matthias > > On 1/16/18 10:08 AM, Colin McCabe wrote: > > Thanks, Matthias, this looks great. > > > > It seems like these APIs could either be used against mock objects, or > against real brokers running in the same process. Is there a way for the > user to select which they want when using the API? Sorry if it's in the > KIP and I missed it. > > > > cheers, > > Colin > > > > > > On Thu, Jan 11, 2018, at 18:06, Matthias J. Sax wrote: > >> Dear Kafka community, > >> > >> I want to propose KIP-247 to add public test utils to the Streams API. > >> The goal is to simplify testing of Kafka Streams applications. > >> > >> Please find details in the wiki: > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams > >> > >> This is an initial KIP, and we hope to add more utility functions later. > >> Thus, this KIP is not comprehensive but a first step. Of course, we can > >> enrich this initial KIP if we think it falls too short. But we should > >> not aim to be comprehensive to keep the scope manageable. > >> > >> In fact, I think we should add some more helpers to simplify result > >> verification. I will update the KIP with this asap. Just wanted to start > >> the discussion early on. > >> > >> An initial WIP PR can be found here: > >> https://github.com/apache/kafka/pull/4402 > >> > >> I also included the user-list (please hit "reply-all" to include both > >> lists in this KIP discussion). > >> > >> Thanks a lot. > >> > >> > >> -Matthias > >> > >> > >> Email had 1 attachment: > >> + signature.asc > >> 1k (application/pgp-signature) > > -- *Saïd Bouras*
Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Colin, the TopologyTestDriver does not connect to any broker and simulates processing of single-partitioned input topics purely in-memory (the driver is basically a mock for a StreamThread). This is sufficient to test basic business logic. For more complex topologies that are actually divided into sub-topologies and connected via topics, the driver detects this case and does an in-memory forward. -Matthias On 1/16/18 10:08 AM, Colin McCabe wrote: > Thanks, Matthias, this looks great. > > It seems like these APIs could either be used against mock objects, or > against real brokers running in the same process. Is there a way for the > user to select which they want when using the API? Sorry if it's in the KIP > and I missed it. > > cheers, > Colin > > > On Thu, Jan 11, 2018, at 18:06, Matthias J. Sax wrote: >> Dear Kafka community, >> >> I want to propose KIP-247 to add public test utils to the Streams API. >> The goal is to simplify testing of Kafka Streams applications. >> >> Please find details in the wiki: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams >> >> This is an initial KIP, and we hope to add more utility functions later. >> Thus, this KIP is not comprehensive but a first step. Of course, we can >> enrich this initial KIP if we think it falls too short. But we should >> not aim to be comprehensive to keep the scope manageable. >> >> In fact, I think we should add some more helpers to simplify result >> verification. I will update the KIP with this asap. Just wanted to start >> the discussion early on. >> >> An initial WIP PR can be found here: >> https://github.com/apache/kafka/pull/4402 >> >> I also included the user-list (please hit "reply-all" to include both >> lists in this KIP discussion). >> >> Thanks a lot. >> >> >> -Matthias >> >> >> Email had 1 attachment: >> + signature.asc >> 1k (application/pgp-signature) signature.asc Description: OpenPGP digital signature
Fwd: Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Forgot dev-list... Forwarded Message Subject: Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams Date: Tue, 16 Jan 2018 13:56:38 -0800 From: Matthias J. Sax <matth...@confluent.io> Organization: Confluent Inc To: users@kafka.apache.org Thanks a lot for the comments. @Guozhang: I updated the KIP accordingly. With regard to potential client test-utils, I agree, but not sure how to resolve it. I guess, we just need to move the class for this case later on. (One reason to annotate all classes with @Evolving) @Bill: The new artifact will be included without the "classifier:test" tag, because it's a regular dependency (the published artifact is not a test artifact). For existing code, we don't remove any existing internal test class in 1.1.0 so the code should still work -- but as internal test classes are internal, we don't provide any guarantee about compatibility in the first place. About `ConsumerRecordFactory`: I think all overloads are useful -- if you remove the overload taking a topicName as input, you cannot overwrite the defaultTopicName and thus need to create a factory for each input topic. On the other hand, if you remove the overloads without talking a topicName, you force people to define a defaultTopicName, and thus they need to create a factory for each topic, too. The goal is to allow the usage of a single factory even if there are multiple topics. The defaultName is useful, if you want to create a lot to record for a single topic, but not a good fit if you create just a few record for each topic. (At least, that is my thinking.). The `null` approach might work, too, but I think this result in ugly boiler plate code and thus I personally prefer the add the overloads. Let me know if you have a strong opinion for the `null` approach with reduced number of overloads. Hope to add the helpers for result verification this week... @Jeff: the available method include global stores -- I added a comment to the KIP -Matthias On 1/16/18 11:28 AM, Jeff Klukas wrote: > From what I can tell, global state stores are managed separately from other > state stores and are accessed via different methods. > > Do the proposed methods on TopologyTestDriver (such as getStateStore) cover > global stores? If not, can we add an interface for accessing and testing > global stores in the scope of this KIP? > > On Thu, Jan 11, 2018 at 9:06 PM, Matthias J. Sax <matth...@confluent.io> > wrote: > >> Dear Kafka community, >> >> I want to propose KIP-247 to add public test utils to the Streams API. >> The goal is to simplify testing of Kafka Streams applications. >> >> Please find details in the wiki: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> 247%3A+Add+public+test+utils+for+Kafka+Streams >> >> This is an initial KIP, and we hope to add more utility functions later. >> Thus, this KIP is not comprehensive but a first step. Of course, we can >> enrich this initial KIP if we think it falls too short. But we should >> not aim to be comprehensive to keep the scope manageable. >> >> In fact, I think we should add some more helpers to simplify result >> verification. I will update the KIP with this asap. Just wanted to start >> the discussion early on. >> >> An initial WIP PR can be found here: >> https://github.com/apache/kafka/pull/4402 >> >> I also included the user-list (please hit "reply-all" to include both >> lists in this KIP discussion). >> >> Thanks a lot. >> >> >> -Matthias >> >> >> > signature.asc Description: OpenPGP digital signature
Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Thanks a lot for the comments. @Guozhang: I updated the KIP accordingly. With regard to potential client test-utils, I agree, but not sure how to resolve it. I guess, we just need to move the class for this case later on. (One reason to annotate all classes with @Evolving) @Bill: The new artifact will be included without the "classifier:test" tag, because it's a regular dependency (the published artifact is not a test artifact). For existing code, we don't remove any existing internal test class in 1.1.0 so the code should still work -- but as internal test classes are internal, we don't provide any guarantee about compatibility in the first place. About `ConsumerRecordFactory`: I think all overloads are useful -- if you remove the overload taking a topicName as input, you cannot overwrite the defaultTopicName and thus need to create a factory for each input topic. On the other hand, if you remove the overloads without talking a topicName, you force people to define a defaultTopicName, and thus they need to create a factory for each topic, too. The goal is to allow the usage of a single factory even if there are multiple topics. The defaultName is useful, if you want to create a lot to record for a single topic, but not a good fit if you create just a few record for each topic. (At least, that is my thinking.). The `null` approach might work, too, but I think this result in ugly boiler plate code and thus I personally prefer the add the overloads. Let me know if you have a strong opinion for the `null` approach with reduced number of overloads. Hope to add the helpers for result verification this week... @Jeff: the available method include global stores -- I added a comment to the KIP -Matthias On 1/16/18 11:28 AM, Jeff Klukas wrote: > From what I can tell, global state stores are managed separately from other > state stores and are accessed via different methods. > > Do the proposed methods on TopologyTestDriver (such as getStateStore) cover > global stores? If not, can we add an interface for accessing and testing > global stores in the scope of this KIP? > > On Thu, Jan 11, 2018 at 9:06 PM, Matthias J. Sax> wrote: > >> Dear Kafka community, >> >> I want to propose KIP-247 to add public test utils to the Streams API. >> The goal is to simplify testing of Kafka Streams applications. >> >> Please find details in the wiki: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> 247%3A+Add+public+test+utils+for+Kafka+Streams >> >> This is an initial KIP, and we hope to add more utility functions later. >> Thus, this KIP is not comprehensive but a first step. Of course, we can >> enrich this initial KIP if we think it falls too short. But we should >> not aim to be comprehensive to keep the scope manageable. >> >> In fact, I think we should add some more helpers to simplify result >> verification. I will update the KIP with this asap. Just wanted to start >> the discussion early on. >> >> An initial WIP PR can be found here: >> https://github.com/apache/kafka/pull/4402 >> >> I also included the user-list (please hit "reply-all" to include both >> lists in this KIP discussion). >> >> Thanks a lot. >> >> >> -Matthias >> >> >> > signature.asc Description: OpenPGP digital signature
Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
>From what I can tell, global state stores are managed separately from other state stores and are accessed via different methods. Do the proposed methods on TopologyTestDriver (such as getStateStore) cover global stores? If not, can we add an interface for accessing and testing global stores in the scope of this KIP? On Thu, Jan 11, 2018 at 9:06 PM, Matthias J. Saxwrote: > Dear Kafka community, > > I want to propose KIP-247 to add public test utils to the Streams API. > The goal is to simplify testing of Kafka Streams applications. > > Please find details in the wiki: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 247%3A+Add+public+test+utils+for+Kafka+Streams > > This is an initial KIP, and we hope to add more utility functions later. > Thus, this KIP is not comprehensive but a first step. Of course, we can > enrich this initial KIP if we think it falls too short. But we should > not aim to be comprehensive to keep the scope manageable. > > In fact, I think we should add some more helpers to simplify result > verification. I will update the KIP with this asap. Just wanted to start > the discussion early on. > > An initial WIP PR can be found here: > https://github.com/apache/kafka/pull/4402 > > I also included the user-list (please hit "reply-all" to include both > lists in this KIP discussion). > > Thanks a lot. > > > -Matthias > > >
Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Thanks for the KIP! One meta question: Will users that are currently using the existing testing code with the "classifier:test" approach: 1) have access to the new testing utilities without updating the gradle.build file 2) can they continue to use the current testing code with the classifier approach? A question on the KIP itself. Since we have factory methods for creating `ConsumerRecord` objects where we can either override or use the default topic do we still all overloads on `ConsumerRecordFactory`? Maybe we could just have constructors specifying the default topic and users could provide "null" if not wanting to specify a topic when creating the `ConsumerRecordFactory`. Even though this is an initial KIP, I agree that adding some helper methods for result verification would be a good idea, but I don't think it should hold up the KIP process if we don't get to it on this pass. Otherwise, it's a +1 from me. Thanks, Bill On Tue, Jan 16, 2018 at 1:01 PM, Guozhang Wangwrote: > Thanks Matthias, I made a pass over the wiki and left some comments; I see > you have addressed most of them. Here are a few more: > > 1. "TopologyTestDriver#process()": how about rename it to "pipeInput" or > "sendInput"? > > 2. For "ConsumerRecordFactory" constructor where "startTimestampMs" is not > specified, what would be the default value? > > > This is not a comment but just reminder: > > 3. ConsumerRecordFactory: I have to admit that putting this class in > o.a.k.streams.test may not be ideal, and if we are going to have an > o.a.k.clients.test in the future testing artifact this may better be moved > there. > > > Guozhang > > > On Mon, Jan 15, 2018 at 2:59 AM, Saïd Bouras > wrote: > > > Hi Matthias, > > > > I read the KIP and it will be very helpful thanks to the changes, I don't > > see though a part that handle topologies that use avro schemas, is it in > > the scope of the KIP ? > > > > I open an issue two month ago in the schema-registry repo : > > https://github.com/confluentinc/schema-registry/issues/651 that explain > > that when testing topologies using schema registry, the schema registry > > client mock is not thread safe and thus in the different processors nodes > > when deserializing it will not work... > > > > In my unit tests I wrapped the mock schema registry client into a > singleton > > but this solution is not enough satisfying. > > > > Thanks in advance, regards :-) > > > > > > On Fri, Jan 12, 2018 at 3:06 AM Matthias J. Sax > > wrote: > > > > > Dear Kafka community, > > > > > > I want to propose KIP-247 to add public test utils to the Streams API. > > > The goal is to simplify testing of Kafka Streams applications. > > > > > > Please find details in the wiki: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 247%3A+Add+public+test+utils+for+Kafka+Streams > > > > > > This is an initial KIP, and we hope to add more utility functions > later. > > > Thus, this KIP is not comprehensive but a first step. Of course, we can > > > enrich this initial KIP if we think it falls too short. But we should > > > not aim to be comprehensive to keep the scope manageable. > > > > > > In fact, I think we should add some more helpers to simplify result > > > verification. I will update the KIP with this asap. Just wanted to > start > > > the discussion early on. > > > > > > An initial WIP PR can be found here: > > > https://github.com/apache/kafka/pull/4402 > > > > > > I also included the user-list (please hit "reply-all" to include both > > > lists in this KIP discussion). > > > > > > Thanks a lot. > > > > > > > > > -Matthias > > > > > > > > > > > > > -- > > > > Saïd BOURAS > > > > Consultant Big Data > > Mobile: 0662988731 > > Zenika Paris > > 10 rue de Milan 75009 Paris > > Standard : +33(0)1 45 26 19 15 <+33(0)145261915> - Fax : +33(0)1 72 70 > 45 > > 10 > > <+33(0)172704510> > > > > > > -- > -- Guozhang >
Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Thanks, Matthias, this looks great. It seems like these APIs could either be used against mock objects, or against real brokers running in the same process. Is there a way for the user to select which they want when using the API? Sorry if it's in the KIP and I missed it. cheers, Colin On Thu, Jan 11, 2018, at 18:06, Matthias J. Sax wrote: > Dear Kafka community, > > I want to propose KIP-247 to add public test utils to the Streams API. > The goal is to simplify testing of Kafka Streams applications. > > Please find details in the wiki: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams > > This is an initial KIP, and we hope to add more utility functions later. > Thus, this KIP is not comprehensive but a first step. Of course, we can > enrich this initial KIP if we think it falls too short. But we should > not aim to be comprehensive to keep the scope manageable. > > In fact, I think we should add some more helpers to simplify result > verification. I will update the KIP with this asap. Just wanted to start > the discussion early on. > > An initial WIP PR can be found here: > https://github.com/apache/kafka/pull/4402 > > I also included the user-list (please hit "reply-all" to include both > lists in this KIP discussion). > > Thanks a lot. > > > -Matthias > > > Email had 1 attachment: > + signature.asc > 1k (application/pgp-signature)
Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Thanks Matthias, I made a pass over the wiki and left some comments; I see you have addressed most of them. Here are a few more: 1. "TopologyTestDriver#process()": how about rename it to "pipeInput" or "sendInput"? 2. For "ConsumerRecordFactory" constructor where "startTimestampMs" is not specified, what would be the default value? This is not a comment but just reminder: 3. ConsumerRecordFactory: I have to admit that putting this class in o.a.k.streams.test may not be ideal, and if we are going to have an o.a.k.clients.test in the future testing artifact this may better be moved there. Guozhang On Mon, Jan 15, 2018 at 2:59 AM, Saïd Bouraswrote: > Hi Matthias, > > I read the KIP and it will be very helpful thanks to the changes, I don't > see though a part that handle topologies that use avro schemas, is it in > the scope of the KIP ? > > I open an issue two month ago in the schema-registry repo : > https://github.com/confluentinc/schema-registry/issues/651 that explain > that when testing topologies using schema registry, the schema registry > client mock is not thread safe and thus in the different processors nodes > when deserializing it will not work... > > In my unit tests I wrapped the mock schema registry client into a singleton > but this solution is not enough satisfying. > > Thanks in advance, regards :-) > > > On Fri, Jan 12, 2018 at 3:06 AM Matthias J. Sax > wrote: > > > Dear Kafka community, > > > > I want to propose KIP-247 to add public test utils to the Streams API. > > The goal is to simplify testing of Kafka Streams applications. > > > > Please find details in the wiki: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 247%3A+Add+public+test+utils+for+Kafka+Streams > > > > This is an initial KIP, and we hope to add more utility functions later. > > Thus, this KIP is not comprehensive but a first step. Of course, we can > > enrich this initial KIP if we think it falls too short. But we should > > not aim to be comprehensive to keep the scope manageable. > > > > In fact, I think we should add some more helpers to simplify result > > verification. I will update the KIP with this asap. Just wanted to start > > the discussion early on. > > > > An initial WIP PR can be found here: > > https://github.com/apache/kafka/pull/4402 > > > > I also included the user-list (please hit "reply-all" to include both > > lists in this KIP discussion). > > > > Thanks a lot. > > > > > > -Matthias > > > > > > > > -- > > Saïd BOURAS > > Consultant Big Data > Mobile: 0662988731 > Zenika Paris > 10 rue de Milan 75009 Paris > Standard : +33(0)1 45 26 19 15 <+33(0)145261915> - Fax : +33(0)1 72 70 45 > 10 > <+33(0)172704510> > -- -- Guozhang
Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams
Hi Matthias, I read the KIP and it will be very helpful thanks to the changes, I don't see though a part that handle topologies that use avro schemas, is it in the scope of the KIP ? I open an issue two month ago in the schema-registry repo : https://github.com/confluentinc/schema-registry/issues/651 that explain that when testing topologies using schema registry, the schema registry client mock is not thread safe and thus in the different processors nodes when deserializing it will not work... In my unit tests I wrapped the mock schema registry client into a singleton but this solution is not enough satisfying. Thanks in advance, regards :-) On Fri, Jan 12, 2018 at 3:06 AM Matthias J. Saxwrote: > Dear Kafka community, > > I want to propose KIP-247 to add public test utils to the Streams API. > The goal is to simplify testing of Kafka Streams applications. > > Please find details in the wiki: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams > > This is an initial KIP, and we hope to add more utility functions later. > Thus, this KIP is not comprehensive but a first step. Of course, we can > enrich this initial KIP if we think it falls too short. But we should > not aim to be comprehensive to keep the scope manageable. > > In fact, I think we should add some more helpers to simplify result > verification. I will update the KIP with this asap. Just wanted to start > the discussion early on. > > An initial WIP PR can be found here: > https://github.com/apache/kafka/pull/4402 > > I also included the user-list (please hit "reply-all" to include both > lists in this KIP discussion). > > Thanks a lot. > > > -Matthias > > > -- Saïd BOURAS Consultant Big Data Mobile: 0662988731 Zenika Paris 10 rue de Milan 75009 Paris Standard : +33(0)1 45 26 19 15 <+33(0)145261915> - Fax : +33(0)1 72 70 45 10 <+33(0)172704510>
[DISCUSS] KIP-247: Add public test utils for Kafka Streams
Dear Kafka community, I want to propose KIP-247 to add public test utils to the Streams API. The goal is to simplify testing of Kafka Streams applications. Please find details in the wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams This is an initial KIP, and we hope to add more utility functions later. Thus, this KIP is not comprehensive but a first step. Of course, we can enrich this initial KIP if we think it falls too short. But we should not aim to be comprehensive to keep the scope manageable. In fact, I think we should add some more helpers to simplify result verification. I will update the KIP with this asap. Just wanted to start the discussion early on. An initial WIP PR can be found here: https://github.com/apache/kafka/pull/4402 I also included the user-list (please hit "reply-all" to include both lists in this KIP discussion). Thanks a lot. -Matthias signature.asc Description: OpenPGP digital signature