Robert Bradshaw created BEAM-12204: -------------------------------------- Summary: Portable Java caches entirety of iterable side inputs. Key: BEAM-12204 URL: https://issues.apache.org/jira/browse/BEAM-12204 Project: Beam Issue Type: Bug Components: java-fn-execution Reporter: Robert Bradshaw Assignee: Robert Bradshaw Fix For: 2.30.0
Though the data is read over the state channel in a paginated way, it is wrapped in LazyCachingIteratorToIterable which forces the entire iterable to be cached. On the other hand, not caching at all for re-iteration could be expensive. We should do what Python does and cache a bit (e.g. the first page) and still support re-iteration by re-requesting the data over the state channel. -- This message was sent by Atlassian Jira (v8.3.4#803005)