[jira] [Commented] (KAFKA-3625) Move kafka-streams test fixtures into a published package
[ https://issues.apache.org/jira/browse/KAFKA-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347850#comment-16347850 ] ASF GitHub Bot commented on KAFKA-3625: --- mjsax opened a new pull request #4502: KAFKA-3625: TopologyTestDriver must process output for wall-clock-time punctuations and on close() URL: https://github.com/apache/kafka/pull/4502 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move kafka-streams test fixtures into a published package > - > > Key: KAFKA-3625 > URL: https://issues.apache.org/jira/browse/KAFKA-3625 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Jeff Klukas >Assignee: Matthias J. Sax >Priority: Minor > Labels: kip, user-experience > Fix For: 1.1.0 > > > The KStreamTestDriver and related fixtures defined in > streams/src/test/java/org/apache/kafka/test would be useful to developers > building applications on top of Kafka Streams, but they are not currently > exposed in a package. > I propose moving this directory to live under streams/fixtures/src/main and > creating a new 'streams:fixtures' project in the gradle configuration to > publish these as a separate package. > KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3625) Move kafka-streams test fixtures into a published package
[ https://issues.apache.org/jira/browse/KAFKA-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346096#comment-16346096 ] ASF GitHub Bot commented on KAFKA-3625: --- mjsax opened a new pull request #4493: KAFKA-3625: add docs for Kafka Streams test-utils (follow up) URL: https://github.com/apache/kafka/pull/4493 *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.* ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move kafka-streams test fixtures into a published package > - > > Key: KAFKA-3625 > URL: https://issues.apache.org/jira/browse/KAFKA-3625 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Jeff Klukas >Assignee: Matthias J. Sax >Priority: Minor > Labels: kip, user-experience > Fix For: 1.1.0 > > > The KStreamTestDriver and related fixtures defined in > streams/src/test/java/org/apache/kafka/test would be useful to developers > building applications on top of Kafka Streams, but they are not currently > exposed in a package. > I propose moving this directory to live under streams/fixtures/src/main and > creating a new 'streams:fixtures' project in the gradle configuration to > publish these as a separate package. > KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3625) Move kafka-streams test fixtures into a published package
[ https://issues.apache.org/jira/browse/KAFKA-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344405#comment-16344405 ] Matthias J. Sax commented on KAFKA-3625: [~guozhang] I linked all relevant JIRAs a could find to this ticket. IMHO, we can only close one of it (I added a corresponding comment to the ticket to get feedback from the reporter). The others still need some work IMHO. > Move kafka-streams test fixtures into a published package > - > > Key: KAFKA-3625 > URL: https://issues.apache.org/jira/browse/KAFKA-3625 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Jeff Klukas >Assignee: Matthias J. Sax >Priority: Minor > Labels: kip, user-experience > Fix For: 1.1.0 > > > The KStreamTestDriver and related fixtures defined in > streams/src/test/java/org/apache/kafka/test would be useful to developers > building applications on top of Kafka Streams, but they are not currently > exposed in a package. > I propose moving this directory to live under streams/fixtures/src/main and > creating a new 'streams:fixtures' project in the gradle configuration to > publish these as a separate package. > KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3625) Move kafka-streams test fixtures into a published package
[
https://issues.apache.org/jira/browse/KAFKA-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320631#comment-16320631
]
Matthias J. Sax commented on KAFKA-3625:
Thanks a lot for this feedback! This is super helpful!
The artifact you are using atm, it not public API, and thus, there is no
guarantee that your tests don't break if you upgrade. (Additionally, you pull
in all Kafka Streams unit tests that you are actually not interested in.) Thus,
we want to have a public {{streams-test-utils}} package. About serialization --
I completely understand that this is annoying, but we cannot easily avoid it...
But we try to minimize the required boilerplate code. Hope you participate in
the KIP discussion that I want to start at the mailing list this week.
> Move kafka-streams test fixtures into a published package
> -
>
> Key: KAFKA-3625
> URL: https://issues.apache.org/jira/browse/KAFKA-3625
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Jeff Klukas
>Assignee: Matthias J. Sax
>Priority: Minor
> Labels: needs-kip, user-experience
>
> The KStreamTestDriver and related fixtures defined in
> streams/src/test/java/org/apache/kafka/test would be useful to developers
> building applications on top of Kafka Streams, but they are not currently
> exposed in a package.
> I propose moving this directory to live under streams/fixtures/src/main and
> creating a new 'streams:fixtures' project in the gradle configuration to
> publish these as a separate package.
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-3625) Move kafka-streams test fixtures into a published package
[ https://issues.apache.org/jira/browse/KAFKA-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320605#comment-16320605 ] Scott Davis commented on KAFKA-3625: FYI I just setup some unit tests using the (not documented) internal test classes in Kafka Streams, and I thought I'd share some thoughts on my experience with it. First, it's super-helpful! The high-level DSL style of Kafka Streams applications doesn't fit well into standard unit testing frameworks, and this solves that problem. It's also helpful compared with integration testing in the sense that it isn't necessary to produce source messages on a broker, wait for timeouts, etc. The internal test classes in 1.0.0 require the unit tests to provide serializers. This can create some extra boilerplate work to configure the serializers (especially the Confluent AVRO Serde, which requires a schema registry). However, I only need to unit test my application logic (i.e. the contents of the "filter" and "map" methods, etc). I can see how testing serializers is a requirement for testing Kafka Streams internally, but it isn't a requirement for testing the logic of Kafka Streams applications. Note: To work around this, I created a "JavaObjectSerde", which uses java.io.Object(In|Out)putStream, for use by the unit tests. I needed about 10 lines of boilerplate code in an @Before method to setup the ProcessorTopologyTestDriver, which seemed slightly excessive but not burdensome. Most of it was to create the serializers and the StreamsConfig, which aren't part of my application logic but were required to create the test driver. > Move kafka-streams test fixtures into a published package > - > > Key: KAFKA-3625 > URL: https://issues.apache.org/jira/browse/KAFKA-3625 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Jeff Klukas >Assignee: Matthias J. Sax >Priority: Minor > Labels: needs-kip, user-experience > > The KStreamTestDriver and related fixtures defined in > streams/src/test/java/org/apache/kafka/test would be useful to developers > building applications on top of Kafka Streams, but they are not currently > exposed in a package. > I propose moving this directory to live under streams/fixtures/src/main and > creating a new 'streams:fixtures' project in the gradle configuration to > publish these as a separate package. > KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-3625) Move kafka-streams test fixtures into a published package
[ https://issues.apache.org/jira/browse/KAFKA-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317097#comment-16317097 ] ASF GitHub Bot commented on KAFKA-3625: --- mjsax opened a new pull request #4402: KAFKA-3625: Add public test utils for Kafka Streams [WIP] URL: https://github.com/apache/kafka/pull/4402 - add new artifact test-utils - add TopologyTestDriver - add MockTime, TestRecord, add TestRecordFactory This PR requires a KIP and is WIP. DO NOT MERGE. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move kafka-streams test fixtures into a published package > - > > Key: KAFKA-3625 > URL: https://issues.apache.org/jira/browse/KAFKA-3625 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Jeff Klukas >Assignee: Matthias J. Sax >Priority: Minor > Labels: needs-kip, user-experience > > The KStreamTestDriver and related fixtures defined in > streams/src/test/java/org/apache/kafka/test would be useful to developers > building applications on top of Kafka Streams, but they are not currently > exposed in a package. > I propose moving this directory to live under streams/fixtures/src/main and > creating a new 'streams:fixtures' project in the gradle configuration to > publish these as a separate package. > KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams -- This message was sent by Atlassian JIRA (v6.4.14#64029)
