GitHub user ppastuszkaocado opened a pull request:
https://github.com/apache/incubator-beam/pull/687
[BEAM-461] Create custom Kinesis source
This adds custom Kinesis source as new beam-sdks-java-io-kinesis module.
Build on Travis has passed.
This has been tested with unit tests (included in the pull request), but
also using DirectRunner (I've confirmed, that source is able to successfully
read data produced by Kinesis Producer Library). I've also tried to run this on
real dataflow, but it doesn't seem to work (the pipeline starts, but it doesn't
read anything at all) - I'm going to investigate the issue, but I'd like you to
do the review in the meantime.
Speaking about tests... I've got bunch of integration / end-to-end level
tests (using DirectRunner and DataflowRunner), which are using real kinesis to
read data from. I'm not sure where should I put them and how would they be
executed in apache CI environment?
BTW. Is this "large" enough, so that I should sign ICLA?
@dhalperi, @jbonofre - can you please review this?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ocadotechnology/incubator-beam
beam_461_kinesis_io
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/687.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #687
----
commit 76b302e1d011e3c1acc9bd2e4e348520f1d7e1d2
Author: Pastuszka PrzemysÅaw <pastuszka.przemys...@gmail.com>
Date: 2016-07-18T18:03:37Z
Moving to apache beam
commit fd6195d32936febe1acf20ea4f8b4a7c4f93f053
Author: PrzemysÅaw Pastuszka <p.pastus...@ocado.com>
Date: 2016-07-19T08:04:09Z
Fix build
commit fb70c6699c95a69a448b3d8d26da0ba381a9e188
Author: PrzemysÅaw Pastuszka <p.pastus...@ocado.com>
Date: 2016-07-19T08:56:02Z
Get rid of dependency on Java 8
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
