tvalentyn commented on code in PR #40283:
URL: https://github.com/apache/beam/pull/40283#discussion_r4125924982
##########
sdks/python/apache_beam/runners/dask/transform_evaluator.py:
##########
@@ -92,8 +92,10 @@ def __iter__(self):
# FIXME(cisaacstern): list() is likely inefficient, since it presumably
# materializes the full result before iterating over it. doing this for
# now as a proof-of-concept. can we can generate results incrementally?
- for result in list(self.bag):
- yield get_windowed_value(result, self.window_fn)
+ # Resolved: we fetch lazily, partition by partition.
+ for partition in self.bag.to_delayed():
Review Comment:
long shot but cc: @alxmrs @cisaacstern in case they still remember some
context around this code or still use this runner :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
