[
https://issues.apache.org/jira/browse/BAHIR-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553577#comment-15553577
]
Christian Kadner edited comment on BAHIR-26 at 10/6/16 11:44 PM:
-----------------------------------------------------------------
As far as I can figure, of the currently five Spark streaming connectors in
Apache Bahir only {{streaming-zeromq}} would require a dockerized test setup.
This is my current assessment:
- {{sql-streaming-mqtt}}: the 3rd-party message broker ActiveMQ is included in
test dependencies
- {{streaming-akka}}: no 3rd-party tools required
- {{streaming-mqtt}}: the 3rd-party message broker ActiveMQ is included in
test dependencies
- {{streaming-twitter}}: the 3rd-party here really is Twitter, I found no
feasible way to isolate a mock Twitter server in a docker container to test
with but we could easily test that a stream can be established and assert
tweets are being consumed (assuming Twitter always has some traffic) or we
could subscribe to our own twitter feed, generate our own tweets and assert we
are consuming them
- {{streaming-zeromq}}: requires native {{zmq}} libraries for test execution,
the test code itself would have to run inside a docker container either by
building and testing the Bahir ZeroMQ connector with maven inside the container
or by mounting the local Bahir and Spark folders into a docker image and
running some tests via Spark submit inside the container and then scrub the
logs for success or failure
was (Author: ckadner):
As far as I can figure, of the currently five Spark streaming connectors in
Apache Bahir only {{streaming-zeromq}} which would require a dockerized test
setup.
This is my current assessment:
- {{sql-streaming-mqtt}}: the 3rd-party message broker ActiveMQ is included in
test dependencies
- {{streaming-akka}}: no 3rd-party tools required
- {{streaming-mqtt}}: the 3rd-party message broker ActiveMQ is included in
test dependencies
- {{streaming-twitter}}: the 3rd-party here really is Twitter, I found no
feasible way to isolate a mock Twitter server in a docker container to test
with but we could easily test that a stream can be established and assert
tweets are being consumed (assuming Twitter always has some traffic) or we
could subscribe to our own twitter feed, generate our own tweets and assert we
are consuming them
- {{streaming-zeromq}}: requires native {{zmq}} libraries for test execution,
the test code itself would have to run inside a docker container either by
building and testing the Bahir ZeroMQ connector with maven inside the container
or by mounting the local Bahir and Spark folders into a docker image and
running some tests via Spark submit inside the container and then scrub the
logs for success or failure
> Add integration test infrastructure based on Docker
> ---------------------------------------------------
>
> Key: BAHIR-26
> URL: https://issues.apache.org/jira/browse/BAHIR-26
> Project: Bahir
> Issue Type: New Feature
> Reporter: Luciano Resende
>
> The existing Spark extensions are all a point of integration with a 3rd party
> tool such as a queue, etc. We want to start doing integration tests that
> actually simulate an environment that indeed communicates with these 3rd
> party tools to make sure we always have an end-to-end test working. In order
> to accomplish that, we could use docker images to launch the 3rd party tool
> involved on the integration tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
