I filed https://issues.apache.org/jira/browse/BEAM-3202
On Thu, Nov 16, 2017 at 9:19 AM, Lukasz Cwik <[email protected]> wrote: > That seems like a bug since its expected that the options id is always set > to something when the PipelineOptions object is created so when > serialized/deserialized the same options id is always returned. > > Seems like a trivial fix in PipelineOptionsFactory to always just call > getOptionsId at least once to ensure it gets populated with a value by the > default value factory. > > On Thu, Nov 16, 2017 at 3:54 AM, Stas Levin <[email protected]> wrote: > >> Hi all, >> >> I'm investigating a memory consumption issue (leak, so it seems) and was >> wondering if it could be related to the way runtime options are handled. >> In particular, upon deserializing a PipelineOptions object, >> ProxyInvocationHandler.Deserializer >> calls ValueProvider.RuntimeValueProvider.setRuntimeOptions(options) which >> stores the (newly) deserialized PipelineOptions instance in a static map >> inside the RuntimeValueProvider class, where the key is an id obtained by >> calling deserializedOptions.getOptionsId(). >> >> The thing is, performing a serialize-deserialize cycle on a given >> PipelineOptions instance and invoking getOptionsId() yields different >> optionsIds. Therefore, multiple deserializations of the same >> PipelineOptions instance result in new keys being added to the static >> "optionsMap" map inside the ValueProvider.RuntimeValueProvider class. >> >> I wasn't able to identify any removals from this static, long-lived map >> (ValueProvider.RuntimeValueProvider#optionsMap), any chance it is >> ever-growing? >> Am I missing something about the way things interact when a given >> PipelineOptions instance gets deserialized numerous times? >> >> Regards, >> Stas >> > >
