Lukasz, It has been a few days since your reply, but I wanted to thank you for pointing me toward the "additional outputs" portion of the documentation. I had already read through that (if not completely thoroughly) although, at the time, I did not quite know the requirements of what I would be doing, so I did not really remember that part. I have some more work to do on my code before I can begin to use Beam (to make it much better!) but I think this should help quite a bit.
Thanks again! Steve On Thu, Sep 12, 2019 at 5:31 PM Lukasz Cwik <lc...@google.com> wrote: > Yes you can create multiple output PCollections using a ParDo with > multiple outputs instead of inserting them into Mongo. > > It could be useful to read through the programming guide related to > PCollections[1] and PTransforms with multiple outputs[2] and feel free to > return with more questions. > > 1: https://beam.apache.org/documentation/programming-guide/#pcollections > 2: > https://beam.apache.org/documentation/programming-guide/#additional-outputs > > On Thu, Sep 12, 2019 at 2:24 PM Steve973 <steve...@gmail.com> wrote: > >> I am new to Beam, and I am pretty excited to get started. I have been >> doing quite a bit of research and playing around with the API. But for my >> use case, unless I am not approaching it correctly, suggests that I will >> need to process multiple PCollections in some parts of my pipeline. >> >> I am working out some of my business logic without a parallelization >> framework to get the solution working. Then I will convert the workflow to >> Beam. What I am doing is reading millions of files from the file system, >> and I am processing parts of the file into three different output types, >> and storing them in MongoDB in three collections. After this initial >> extraction (mapping), I modify some of the data which will result in >> duplicates. So the next step is a reduction step to eliminate the >> duplicates (based on a number of fields) and aggregate the references to >> the other 2 data types, so the reduced object contains the dedupe fields, >> and a list of references to documents in the other 2 collections. I'm not >> touching either of these two collections at this time, but this is where my >> question comes in. If I map this data, can I create three separate >> PCollections instead of inserting them into Mongo? After the >> deduplication, I will need to combine data in two of the streams, and I >> need to store the results of that combination into mongo. Then I need to >> process the third collection, which will go into its own mongo collection. >> >> I hope my description was at least enough to get the conversation >> started. Is my approach reasonable, and can I create multiple PCollections >> and use them at different phases of my pipeline? Or is there another way >> that I should be looking at this? >> >> Thanks in advance! >> Steve >> >
