[jira] [Commented] (KAFKA-2377) Add copycat system tests
[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717506#comment-14717506 ] ASF GitHub Bot commented on KAFKA-2377: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/150 Add copycat system tests Key: KAFKA-2377 URL: https://issues.apache.org/jira/browse/KAFKA-2377 Project: Kafka Issue Type: Sub-task Components: copycat Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava Fix For: 0.8.3 Add baseline system tests for Copycat, covering both standalone and distributed mode. This should cover basic failure modes and verify at-least-one delivery of data, both from source system - Kafka and Kafka - sink system. This, of course, requires testing the core, built-in connectors provided with Copycat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2377) Add copycat system tests
[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710668#comment-14710668 ] Ewen Cheslack-Postava commented on KAFKA-2377: -- By the way, might be worth pointing out that current trunk will have some failing/hanging system tests due to KAFKA-2467. If you want to run the full test suite you'll need the patch for that issue as well. Add copycat system tests Key: KAFKA-2377 URL: https://issues.apache.org/jira/browse/KAFKA-2377 Project: Kafka Issue Type: Sub-task Components: copycat Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava Fix For: 0.8.3 Add baseline system tests for Copycat, covering both standalone and distributed mode. This should cover basic failure modes and verify at-least-one delivery of data, both from source system - Kafka and Kafka - sink system. This, of course, requires testing the core, built-in connectors provided with Copycat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2377) Add copycat system tests
[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702585#comment-14702585 ] ASF GitHub Bot commented on KAFKA-2377: --- GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/150 KAFKA-2377: Add basic system test for copycat using source and sink file connectors. Tests standalone mode by running separate source and sink connectors, catting data into the source file, and validating the output in the sink file. Restarts the service to verify that clean restarts will result in tasks resuming where they left off. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ewencp/kafka kafka-2377-copycat-system-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/150.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #150 commit 10f6c17963b585c8430c491f836f13c09c00a5ec Author: Ewen Cheslack-Postava m...@ewencp.org Date: 2015-08-14T21:08:13Z KAFKA-2377: Add basic system test for copycat using source and sink file connectors. Tests standalone mode by running separate source and sink connectors, catting data into the source file, and validating the output in the sink file. Restarts the service to verify that clean restarts will result in tasks resuming where they left off. Add copycat system tests Key: KAFKA-2377 URL: https://issues.apache.org/jira/browse/KAFKA-2377 Project: Kafka Issue Type: Sub-task Components: copycat Reporter: Ewen Cheslack-Postava Fix For: 0.8.3 Add baseline system tests for Copycat, covering both standalone and distributed mode. This should cover basic failure modes and verify at-least-one delivery of data, both from source system - Kafka and Kafka - sink system. This, of course, requires testing the core, built-in connectors provided with Copycat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2377) Add copycat system tests
[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642336#comment-14642336 ] Gwen Shapira commented on KAFKA-2377: - Two connectors that are not useful for anything except testing, but are very useful for testing: * sequence source (i.e generate a number ever X ms and send to Kafka) * log sink (i.e. write to a log what it is reading from Kafka. The equivalent of a console consumer) This is independent of testing the connectors, but IMO it makes sense. Add copycat system tests Key: KAFKA-2377 URL: https://issues.apache.org/jira/browse/KAFKA-2377 Project: Kafka Issue Type: Sub-task Reporter: Ewen Cheslack-Postava Fix For: 0.8.3 Add baseline system tests for Copycat, covering both standalone and distributed mode. This should cover basic failure modes and verify at-least-one delivery of data, both from source system - Kafka and Kafka - sink system. This, of course, requires testing the core, built-in connectors provided with Copycat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2377) Add copycat system tests
[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642367#comment-14642367 ] Ewen Cheslack-Postava commented on KAFKA-2377: -- Good point -- I think this is similar to what we need for regular Kafka system tests too. The sequence source you describe is similar to the VerifiableProducer that [~granders] has worked on. One thing that's more difficult about testing Copycat is that the log data can be spread across many hosts in distributed mode if the task ends up running across different workers. On the sink side, I think the patch for the initial version of Copycat in KAKFA-2666 already addresses that since it has a file sink, though again, distributed mode adds a bit of a wrinkle testing-wise. Add copycat system tests Key: KAFKA-2377 URL: https://issues.apache.org/jira/browse/KAFKA-2377 Project: Kafka Issue Type: Sub-task Reporter: Ewen Cheslack-Postava Fix For: 0.8.3 Add baseline system tests for Copycat, covering both standalone and distributed mode. This should cover basic failure modes and verify at-least-one delivery of data, both from source system - Kafka and Kafka - sink system. This, of course, requires testing the core, built-in connectors provided with Copycat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
