FYI, I just checked in WriteResult.getSuccessfulStorageApiInserts, which should give you what you need. This is now checked into Beam HEAD, and should be included in the next Beam release.
On Fri, Mar 3, 2023 at 12:51 PM Matthew Ouyang <matthew.ouy...@gmail.com> wrote: > I'm currently not using Wait.on. I have a pipeline that uses BigQuery for > checkpointing purposes. I only want records that are > successfully checkpointed in BigQuery to be eligible for the next stage in > my pipeline. With streaming inserts, I can use getSuccessfulInserts to > achieve this and I was looking for something similar with Storage Write. > > On Thu, Mar 2, 2023 at 4:48 PM Reuven Lax via user <user@beam.apache.org> > wrote: > >> Are you trying to do this in order to use Wait.on? getSuccessfulInserts >> is not currently supported for Storage Write API. >> >> On Thu, Mar 2, 2023 at 1:44 PM Matthew Ouyang <matthew.ouy...@gmail.com> >> wrote: >> >>> Thank you to Ahmed and Reuven for the tip on >>> WriteResult::getFailedStorageApiInserts. >>> >>> When I tried to get the successful inserts through the Storage Write >>> API, I received an error message saying that "Retrieving successful inserts >>> is only supported for streaming inserts. Make sure >>> withSuccessfulInsertsPropagation is correctly configured for >>> BigQueryIO.Write object." Did I make a mistake, or is there a reason why >>> this is not possible? I tried setting triggeringFrequency + >>> numStorageWriteApiStreams as required by Storage Write, and I tried to set >>> successfulInsertsPropagation as directed in the error message. >>> >>
