InigoSJ commented on a change in pull request #15106:
URL: https://github.com/apache/beam/pull/15106#discussion_r661774465
##########
File path: sdks/python/apache_beam/io/fileio.py
##########
@@ -243,6 +249,57 @@ def process(
yield ReadableFile(metadata, self._compression)
+@experimental()
+class MatchContinuously(beam.PTransform):
+ """Checks for new files for a given pattern every interval.
+
+ This ``PTransform`` returns a ``PCollection`` of matching files in the form
+ of ``FileMetadata`` objects.
Review comment:
I thought about, but I am not sure this would have an actual use case.
Not allowing EmptyMatches would produce an error, and since this is a streaming
pipeline, that errors would be retried infinitely.
https://beam.apache.org/releases/pydoc/2.30.0/_modules/apache_beam/io/fileio.html#EmptyMatchTreatment
`MatchAll` has the option to add it and wouldn't be hard, but I think it
would translate in the PTransform being less resilient. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
