I think full-blown SDF is not needed for this - someone just needs to implement a MongoDbIO.readAll() variant, using a composite transform. The regular pattern for this sort of thing will do (ParDo split, reshuffle, ParDo read). Whether it's worth replacing MongoDbIO.read() with a redirect to readAll() is another matter - size estimation, available only in BoundedSource for now, may or may not be important.
On Wed, Jul 17, 2019 at 2:39 AM Ryan Skraba <r...@skraba.com> wrote: > Hello! To clarify, you want to do something like this? > > PubSubIO.read() -> extract mongodb collection and range -> > MongoDbIO.read(collection, range) -> ... > > If I'm not mistaken, it isn't possible with the implementation of > MongoDbIO (based on BoundedSource interface, requiring the collection to be > specified once at pipeline construction time). > > BUT -- this is a good candidate for an improvement in composability, and > the ongoing work to prefer the SDF for these types of use cases. Maybe > raise a JIRA for an improvement? > > All my best, Ryan > > > On Wed, Jul 17, 2019 at 9:35 AM Chaim Turkel <ch...@behalf.com> wrote: > >> any ideas? >> >> On Mon, Jul 15, 2019 at 11:04 PM Rui Wang <ruw...@google.com> wrote: >> > >> > +user@beam.apache.org >> > >> > >> > -Rui >> > >> > On Mon, Jul 15, 2019 at 6:55 AM Chaim Turkel <ch...@behalf.com> wrote: >> >> >> >> Hi, >> >> I am looking to write a pipeline that read from a mongo collection. >> >> I would like to listen to a pubsub that will have a object that will >> >> tell me which collection and which time frame. >> >> Is there a way to do this? >> >> >> >> Chaim >> >> >> >> -- >> >> >> >> >> >> Loans are funded by >> >> FinWise Bank, a Utah-chartered bank located in Sandy, >> >> Utah, member FDIC, Equal >> >> Opportunity Lender. Merchant Cash Advances are >> >> made by Behalf. For more >> >> information on ECOA, click here >> >> <https://www.behalf.com/legal/ecoa/>. For important information about >> >> opening a new >> >> account, review Patriot Act procedures here >> >> <https://www.behalf.com/legal/patriot/>. >> >> Visit Legal >> >> <https://www.behalf.com/legal/> to >> >> review our comprehensive program terms, >> >> conditions, and disclosures. >> >> -- >> >> >> Loans are funded by >> FinWise Bank, a Utah-chartered bank located in Sandy, >> Utah, member FDIC, Equal >> Opportunity Lender. Merchant Cash Advances are >> made by Behalf. For more >> information on ECOA, click here >> <https://www.behalf.com/legal/ecoa/>. For important information about >> opening a new >> account, review Patriot Act procedures here >> <https://www.behalf.com/legal/patriot/>. >> Visit Legal >> <https://www.behalf.com/legal/> to >> review our comprehensive program terms, >> conditions, and disclosures. >> >
