Abacn commented on code in PR #39529:
URL: https://github.com/apache/beam/pull/39529#discussion_r3707392608
##########
website/www/site/content/en/documentation/io/developing-io-python.md:
##########
@@ -69,23 +69,39 @@ for Beam's transform style guidance.
To create a new data source for your pipeline, you'll need to provide the
format-specific logic that tells the service how to read data from your input
source, and how to split your data source into multiple parts so that multiple
worker instances can read your data in parallel.
+If you're creating a data source that reads unbounded data, you must also
+provide the logic for managing your source's watermark and checkpointing.
+
Supply the logic for your new source by creating the following classes:
- * A subclass of `BoundedSource`. `BoundedSource` is a source that reads a
- finite amount of input records. The class describes the data you want to
- read, including the data's location and parameters (such as how much data
to
+ * A subclass of `BoundedSource` if you want to read a finite (batch) data
set,
+ or a subclass of `UnboundedSource` if you want to read an infinite
+ (streaming) data set. The class describes the data you want to read,
+ including the data's location and parameters (such as how much data to
read).
- * A subclass of `RangeTracker`. `RangeTracker` is a thread-safe object used
to
- manage a range for a given position type.
+ * For a `BoundedSource`, a subclass of `RangeTracker`. `RangeTracker` is a
+ thread-safe object used to manage a range for a given position type.
+ * For an `UnboundedSource`, a subclass of `UnboundedReader`, which holds the
+ state involved in reading the stream, and a subclass of `CheckpointMark`,
+ which records the position that a reader resumes from.
* One or more user-facing wrapper composite transforms (`PTransform`) that
wrap read operations. [PTransform wrappers](#ptransform-wrappers) discusses
why you should avoid exposing your sources, and walks through how to create
a wrapper.
-You can find these classes in the
-[apache_beam.io.iobase module](https://beam.apache.org/releases/pydoc/{{<
param release_latest >}}/apache_beam.io.iobase.html).
+You can find `BoundedSource` and `RangeTracker` in the
+[apache_beam.io.iobase module](https://beam.apache.org/releases/pydoc/{{<
param release_latest >}}/apache_beam.io.iobase.html),
+and the unbounded classes in the
+[apache_beam.io.unbounded_source
module](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/unbounded_source.py).
Review Comment:
Let's use https://beam.apache.org/releases/pydoc/ website. This PR won't be
merged until Beam 2.76.0 released, at that time pydoc website is updated to
have unbounded_source
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
