knockknock10 commented on issue #34419: URL: https://github.com/apache/beam/issues/34419#issuecomment-6826676767
> Hi, yes that is the case I am concerned about as well. I think that it was decided to not pursue Beam 3.0 as a big bundling of improvements/features and instead to make progress gradually and to instead announce such changes to behavior. > > So yes, you'd be welcome to help contribute here. Writing a test sounds like a good way to start. As for how to capture, I was thinking something along the lines of wiring output from A to go through a reciever that has a try-catch block around forwarding to B. Then if B throws an exception the receiver could note it. We could have various modes controlled by options on what the receiver does then: > > 1. throw the exception, same as today > 2. throw the exception and if bundle would otherwise complete successfully log a warning that the error was swallowed by fusion > 3. throw the exception, and if the bundle would otehrwise complete successfully prevent it. We could also attempt to abort the bundle so we don't process further since it will be wasted work. Thanks, Sam! I went through the issue and started looking into how the exception is passed between fused DoFns. From what I understand, when DoFn A sends an element to DoFn B and B throws an exception, that exception travels back to A. If A catches it, the bundle might still complete successfully even though B failed. I think we should first reproduce this properly with a test. I'll start by checking the existing Java harness tests and add a regression test with two fused DoFns, where A catches the exception from B and continues processing. I'll also check the case where A doesn't catch the exception to make sure the existing behaviour is covered. Once we have the test reproducing the issue, I'll look into the receiver wrapper approach you suggested and how we can track such failures at the bundle level. I'll also check how this works with bundle completion and processor reuse before deciding where the change should go. I'll start with the tests and share what I find here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
