Hi Jan, Thanks a lot for the reply! This helps, I wonder if you have any idea whats the difference between fasterCopy vs objectReuse option?
Eleanore On Fri, Apr 9, 2021 at 11:53 AM Jan Lukavský <je...@seznam.cz> wrote: > Hi Eleanore, > > the --fasterCopy option disables clone between operators (see [1]). It > should be safe to use it, unless your pipeline outputs an object and > later modifies the same instance. This is generally not supported by the > Beam model and is considered to be an user error. FlinkRunner > historically chose a way of "better-safe-than-sorry" approach and > explicitly cloned every received object between (non-shuffle) operators. > Enabling this option should increase performance, you can verify your > Pipeline is not doing any disallowed mutations using DirectRunner, which > checks this by default (without --enforceImmutability=false). > > Jan > > [1] https://issues.apache.org/jira/browse/BEAM-11146 > > On 4/9/21 7:57 AM, Eleanore Jin wrote: > > Hi community, > > > > I am upgrading from Beam 2.23.0 -> 2.28.0, and a new > > FlinkPipelineOption is introduced: fasterCopy. > > > > Can you please help me understand what is the difference between the > > option objectReuse vs fasterCopy? > > > > Thanks a lot! > > Eleanore >
