[
https://issues.apache.org/jira/browse/BEAM-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383977#comment-15383977
]
ASF GitHub Bot commented on BEAM-461:
-------------------------------------
GitHub user ppastuszkaocado opened a pull request:
https://github.com/apache/incubator-beam/pull/687
[BEAM-461] Create custom Kinesis source
This adds custom Kinesis source as new beam-sdks-java-io-kinesis module.
Build on Travis has passed.
This has been tested with unit tests (included in the pull request), but
also using DirectRunner (I've confirmed, that source is able to successfully
read data produced by Kinesis Producer Library). I've also tried to run this on
real dataflow, but it doesn't seem to work (the pipeline starts, but it doesn't
read anything at all) - I'm going to investigate the issue, but I'd like you to
do the review in the meantime.
Speaking about tests... I've got bunch of integration / end-to-end level
tests (using DirectRunner and DataflowRunner), which are using real kinesis to
read data from. I'm not sure where should I put them and how would they be
executed in apache CI environment?
BTW. Is this "large" enough, so that I should sign ICLA?
@dhalperi, @jbonofre - can you please review this?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ocadotechnology/incubator-beam
beam_461_kinesis_io
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/687.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #687
----
commit 76b302e1d011e3c1acc9bd2e4e348520f1d7e1d2
Author: Pastuszka Przemysław <pastuszka.przemys...@gmail.com>
Date: 2016-07-18T18:03:37Z
Moving to apache beam
commit fd6195d32936febe1acf20ea4f8b4a7c4f93f053
Author: Przemysław Pastuszka <p.pastus...@ocado.com>
Date: 2016-07-19T08:04:09Z
Fix build
commit fb70c6699c95a69a448b3d8d26da0ba381a9e188
Author: Przemysław Pastuszka <p.pastus...@ocado.com>
Date: 2016-07-19T08:56:02Z
Get rid of dependency on Java 8
----
> Create custom Kinesis source
> ----------------------------
>
> Key: BEAM-461
> URL: https://issues.apache.org/jira/browse/BEAM-461
> Project: Beam
> Issue Type: Improvement
> Affects Versions: 0.2.0-incubating
> Reporter: Przemyslaw Pastuszka
> Labels: newbie
>
> Design doc is here:
> https://docs.google.com/document/d/1O_m9albjxzBdGf94gbU1682kSwGEKjsQ0AMF5FBsjUU/edit?usp=sharing
> This has been already proposed for contribution to DataflowJavaSDK:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/pull/149
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
