damccorm commented on code in PR #23830:
URL: https://github.com/apache/beam/pull/23830#discussion_r1010830178
##########
sdks/python/apache_beam/ml/inference/base_test.py:
##########
@@ -171,6 +171,38 @@ def test_unexpected_inference_args_passed(self):
FakeModelHandlerFailsOnInferenceArgs(),
inference_args=inference_args)
+ def test_increment_failed_batches_counter(self):
+ with self.assertRaises(ValueError, FakeModelHandlerFailsOnInferenceArgs):
+ with TestPipeline() as pipeline:
+ examples = [1, 5, 3, 10]
+ pcoll = pipeline | 'start' >> beam.Create(examples)
+ inference_args = {'key': True}
+ _ = pcoll | base.RunInference(FakeModelHandlerFailsOnInferenceArgs(),
+ inference_args=inference_args)
+ run_result = pipeline.run()
+ run_result.wait_until_finish()
+
+ metric_results = (
+
run_result.metrics().query(MetricsFilter().with_name('failed_batches_counter')))
+ num_failed_batches_counter = metric_results['counters'][0]
+ self.assertEqual(num_failed_batches_counter.committed, 1)
Review Comment:
> And it seems like you're saying that if the inference fails, the test will
retry the batch a certain number of times. If that's consistent, we could force
the batching to make sure we have a reliable count. For the moment I've just
changed it to assert that the failed_batches_counter will be greater than zero,
but I could make that more precise if you can confirm my interpretation of what
you're saying.
I'm not 100% sure what the counter behavior is, but yeah it should retry the
batch 3 times; your understanding is correct
> I'm not totally following what you're saying about transient failures
though. Do you mean, set up a FakeModelHandler to fail the first time and then
succeed, to see if we count the failure?
Yeah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
