ajdub508 commented on code in PR #28091:
URL: https://github.com/apache/beam/pull/28091#discussion_r1313806276
##########
sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py:
##########
@@ -454,6 +454,56 @@ def test_big_query_write_insert_errors_reporting(self):
| 'ParseErrors' >> beam.Map(lambda err: (err[1], err[2])),
equal_to(bq_result_errors))
+
+ @pytest.mark.it_postcommit
+ def test_big_query_write_insert_not_found_errors(self):
+ """
+ Test that NotFound errors returned by beam.io.WriteToBigQuery
+ contain both the failed rows amd the reason for it failing.
Review Comment:
Thanks for the feedback, @ahmedabu98. I think we could raise 429, 500, and
503 errors to let the pipeline errors be retried indefinitely and route all the
others to the failed rows tag. This would be consistent with [errors defined as
transient in API Core
Retry](https://github.com/googleapis/python-api-core/blob/main/google/api_core/retry.py#L113-L135)
and [AIP-194](https://google.aip.dev/194)
Those retryable errors would all be caught in the first except clause. The
second clause catching
[HttpError](https://github.com/google/apitools/blob/master/apitools/base/py/exceptions.py#L111-L117)
type doesn't contain anything that is typically considered transient and
they'll go straight to failed rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
