Re: Side outputs from sinks

2023-10-18 Thread Péter Váry
Hi Aian, Which sink API are you using? Have you tried the Sink v2 API [1]? If you implement the WithPostCommitTopology interface [2], then you can provide a follow-up step after the commits are finished. I have not tried yet, but I expect that the failed Committables are emitted as well, and

Side outputs from sinks

2023-10-18 Thread Aian Cantabrana
Hi, We have an use case where we need to ensure that data reaches all endpoints/sinks one by one or to split the flow if any of them fails. Here is an schema of the use case: Current job: Source -> filters/maps/process -> sink1 \-> sink2 \-> sink3 Desired job: Source ->