That's the evolution I'm proposing and I already implemented in some IO: readAll pattern. Let me check for mongo.
On Oct 30, 2017, 12:00, at 12:00, Chaim Turkel <[email protected]> wrote: >I am syncing multiple tables from mongo to bigquery. >So i first check how many records there are, and then if there are >records a need to sync them, else i need to update the status table, >that there was nothing to sync. Also in the case that I do sync i need >to update the status table with information about the sync. > >Why can't the read start from a collection also? >chaim > >On Mon, Oct 30, 2017 at 12:13 PM, Jean-Baptiste Onofré ><[email protected]> wrote: >> Can you describe your use case ? We can imagine to be able to define >a custom FN in the read. But I'm afraid it would be too specific. >> >> On Oct 30, 2017, 10:46, at 10:46, Chaim Turkel <[email protected]> >wrote: >>>any reason for this, there should be a way to run it from any point >>> >>>On Mon, Oct 30, 2017 at 11:24 AM, Jean-Baptiste Onofré >>><[email protected]> wrote: >>>> Hi >>>> >>>> No the pipeline starts with the read. You can always create your >own >>>custom read. >>>> >>>> Regards >>>> JB >>>> >>>> On Oct 30, 2017, 09:33, at 09:33, Chaim Turkel <[email protected]> >>>wrote: >>>>>Hi, >>>>> Is there a way to have some code run before the read? >>>>>I would like to check before how many records exists and based on >>>this >>>>>have two different pipelines. >>>>>Currently this code is in the runner but since i have 20 tables >this >>>>>takes a long time. >>>>>I would like to move the check into the pipeline - >>>>> >>>>>any ideas? >>>>> >>>>> >>>>>chaim
