This makes good sense and completes the picture for me. If I still wanted to do in place aggregation, I could use the Aggregator interface which would then close off the persistentAggregate option. In this particular instance I'm actually doing the aggregation in a DRPC, downstream from a state query, so that would be a viable choice, although coming at the cost of being able to do partial aggregation before network transfer...the in-place merging is probably a premature optimization, but it also simplified my code so I'm glad I understand why it is a bad idea.
Thanks. On Sun, Dec 22, 2013 at 5:16 AM, Svend Vanderveken < [email protected]> wrote: > > Here's a situation when doing so leads to an issue: when combining the > aggregator with an opaque state, doing an in place update in the aggregator > implies that the "previous" and "current" references point to the same java > instance, so essentially the previous value is lost and we lose the > robustness of the opaque approach. > On Dec 22, 2013 3:58 AM, "Nathan Marz" <[email protected]> wrote: > >> Actually, nevermind, those cases aren't problems since these objects are >> contained within the operation of the combiner aggregator. But I still >> wouldn't do it as it goes against the spirit of the API. >> >> >> On Sat, Dec 21, 2013 at 6:56 PM, Nathan Marz <[email protected]>wrote: >> >>> They could potentially be re-used if the stream forks at that point or >>> if there are multiple aggregators chained together. >>> >>> >>> On Sat, Dec 21, 2013 at 4:24 PM, Adam Lewis <[email protected]> wrote: >>> >>>> Okay...what I've suggested isn't the cleanest seeming and I'm happy to >>>> move to a less...smelly solution, but I'm unclear on how those values would >>>> get reused. I'm not touching objects pointed to by the original tuple, >>>> only mutating the object created in the init method of the >>>> CombinerAggregator. >>>> >>>> Just to better my understanding of how storm/trident works, in what >>>> circumstances could those values be reused? >>>> >>>> Thanks. >>>> >>>> >>>> On Sat, Dec 21, 2013 at 5:49 PM, Nathan Marz <[email protected]>wrote: >>>> >>>>> You shouldn't do that. Those input values could potentially be re-used >>>>> for other operations. I would recommend looking into using persistent data >>>>> structures. >>>>> >>>>> >>>>> On Sat, Dec 21, 2013 at 6:42 AM, Adam Lewis <[email protected]>wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> When implementing CombinerAggregator in Trident, >>>>>> is >>>>>> there any issue with aggregating in >>>>>> place? That is, something like this: >>>>>> >>>>>> >>>>>> public class >>>>>> MyAgg >>>>>> implements CombinerAggregator< >>>>>> MyMutable >>>>>> > { >>>>>> >>>>>> >>>>>> public MyMutable combine(MyMutable val1, MyMutable val2) { >>>>>> >>>>>> val1.mergeInPlace(val2); >>>>>> return val1; >>>>>> >>>>>> } >>>>>> >>>>>> ... >>>>>> >>>>>> } >>>>>> >>>>>> In my use case the values are somewhat elaborate structurally, and >>>>>> combining them in place is a lot more efficient...this seems to be >>>>>> working >>>>>> fine in local mode, but I can't find any indication if this is a sound >>>>>> practice before I commit to it. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Adam >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Twitter: @nathanmarz >>>>> http://nathanmarz.com >>>>> >>>> >>>> >>> >>> >>> -- >>> Twitter: @nathanmarz >>> http://nathanmarz.com >>> >> >> >> >> -- >> Twitter: @nathanmarz >> http://nathanmarz.com >> >
