If you want to wait for all records are written (per window) to Cassandra
before writing that window to PubSub, you should be able to use the Wait
transform:
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Wait.java

Thanks,
Cham

On Wed, Mar 24, 2021 at 9:49 AM Alexey Romanenko <aromanenko....@gmail.com>
wrote:

> Do you want to wait for ALL records are written for Cassandra and then
> write all successfully written records to PubSub or it should be performed
> "record by record"?
>
> On 24 Mar 2021, at 04:58, Vincent Marquez <vincent.marq...@gmail.com>
> wrote:
>
> I have a common use case where my pipeline looks like this:
> CassandraIO.readAll -> Aggregate -> CassandraIO.write -> PubSubIO.write
>
> I do NOT want my pipeline to look like the following:
>
> CassandraIO.readAll -> Aggregate -> CassandraIO.write
>                                                          |
>                                                           -> PubsubIO.write
>
> Because I need to ensure that only items written to Pubsub have
> successfully finished a (quorum) write.
>
> Since CassandraIO.write is a PTransform<A, PDone> I can't actually use it
> here so I often roll my own 'writer', but maybe there is a recommended way
> of doing this?
>
> Thanks in advance for any help.
>
> *~Vincent*
>
>
>

Reply via email to