[
https://issues.apache.org/jira/browse/BEAM-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268140#comment-16268140
]
nevi_me edited comment on BEAM-2639 at 11/28/17 5:25 AM:
---------------------------------------------------------
MongoDB 3.6 will include a `$changeStream` operator, enabling unbounded access
to collections without tailing the oplog.
[http://mongodb.github.io/mongo-java-driver/3.6/driver/tutorials/change-streams/]
I'm busy preparing to upgrade to MongoDB, I tail the Oplog a lot, and want to
implement these change streams. I'd like to contribute an unbounded source for
MongoDB, I think it'll help me learn more about Beam's internals. It's not a
high priority, but since it'd be my first contribution to ASF, I might need
some hand-holding when the time comes.
The one thing I need to first investigate is if there are breaking changes to
the way users authenticate to databases. Some drivers have been logging
deprecation warnings about upcoming 3.6 changes. I'll also look into that.
I'll provide feedback mid-December when I have downtime.
was (Author: nevi_me):
MongoDB 3.6 will include a `$changeStream` operator, enabling unbounded access
to collections.
[http://mongodb.github.io/mongo-java-driver/3.6/driver/tutorials/change-streams/]
I'm busy preparing to upgrade to MongoDB, I tail the Oplog a lot, and want to
implement these change streams. I'd like to contribute an unbounded source for
MongoDB, I think it'll help me learn more about Beam's internals. It's not a
high priority, but since it'd be my first contribution to ASF, I might need
some hand-holding when the time comes.
The one thing I need to first investigate is if there are breaking changes to
the way users authenticate to databases. Some drivers have been logging
deprecation warnings about upcoming 3.6 changes. I'll also look into that.
I'll provide feedback mid-December when I have downtime.
> Unbounded Source for MongoDB
> ----------------------------
>
> Key: BEAM-2639
> URL: https://issues.apache.org/jira/browse/BEAM-2639
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
> Affects Versions: 2.0.0
> Reporter: nevi_me
> Assignee: Jean-Baptiste Onofré
> Priority: Minor
>
> The current MongoDB source is bounded, which means that we can't build
> streaming pipelines directly from MongoDB.
> MongoDB publishes changes in each collection through the oplog. Would it be
> possible to create a connector that reads the oplog to create an unbounded
> source?
> As an oplog is only available through replication, this creates that
> dependency. We would need to also consider whether a polling method (using
> the ObjectId) could be an appropriate fallback.
> Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
