[
https://issues.apache.org/jira/browse/BEAM-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259508#comment-16259508
]
Kenneth Knowles commented on BEAM-3227:
---------------------------------------
At the moment, we don't really have any examples other than Combine (which is
arguably the most important composite; nearly a primitive) so I don't know what
might be typical. I can certainly imagine a composite having multiple UDFs with
different environments, though it would probably mean that it is built from
primitives with UDFs in different environments. This is a high generalized
capability that we may never use, but is easy enough to have via multiple
fields that are pointers to environments in the payload.
> Consider sharing Udf/SkdFunctionSpec records via pointer
> --------------------------------------------------------
>
> Key: BEAM-3227
> URL: https://issues.apache.org/jira/browse/BEAM-3227
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
> Reporter: Kenneth Knowles
>
> Coders are stored by pointer, because they are often repeated and a common
> source of huge pipeline descriptions.
> We considered doing the same for all UDFs but decided not to, based on the
> logic that they are not as often identical and will rarely implement the
> equals() needed to actually share encoded versions.
> However, in the presence of generated code, it is very likely that DoFns and
> CombineFns are repeated, and also much more likely that they have meaningful
> equals(), so there could be size savings.
> None of this is terribly important for storage or transmission, but has more
> to do with arbitrary and small size limits that occur in some API frameworks
> or database column types.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
