Hi Daniel,

What about creating your Redis Key in format YYYYMMDDHHmm_DeviceId or may
be just HHmm_DeviceId

i.e.
201604111213_A or 1213_A

and give the key expiry time as 2 minutes. i assume you dont care about
past data.

Now whenevr a new value come set this key Value (YYYYMMDDHHmm_DeviceId or
HHmm_DeviceId)

and on aggregation side whenever you have to add all values, just get the
list of all of ur devices, build all keys with ur format and do a multiget
for all keys i.e 201604111213_A, 201604111213_B etc and add it. I think
redis also has a way to do the sum and save the result into other key.


Thanks
Ravi.



On Sun, Apr 10, 2016 at 9:46 PM, Daniela Stoiber <daniela.stoi...@gmail.com>
wrote:

> HI Arun,
>
> thank you for your reply.
>
> But my problem is that I need to add up the values over all devices, but I
> am only allowed to use the most recent value of each device. A value is
> valid as long as there is no new value for this device available.
>
> So if I receive a message with device A with value 1, value 1 should be
> used for the sum as long as the value of A does not change.
> When I receive a new value for A, the new value should be used for the sum
> and the old one should be replaced.
>
> Therefore I thought to use Redis to store this information:
> Device          Value
> A               1
> B               10
> C               4
>
> Then I would like to pull every minute the most recent value of each
> device to build the sum. Therefore I would like to use the windowed bolt.
> But I am not sure if it is possible to pull data out of Redis within a
> windowed bolt.
>
> Thank you in advance.
>
> Regards,
> Daniela
>
>
> -----Ursprüngliche Nachricht-----
> Von: Arun Iyer [mailto:ai...@hortonworks.com] Im Auftrag von Arun
> Mahadevan
> Gesendet: Sonntag, 10. April 2016 20:55
> An: dev@storm.apache.org
> Betreff: Re: AW: Use only latest values
>
> Hi Matthias,
>
> WindowedBolt does support event time. In trident its is not yet exposed.
>
> Hi Daniela,
>
> You could solve your use cases in different ways. One would be to have a
> WindowedBolt with a 1 min tumbling window, do your custom aggregation (e.g.
> sum) every time the window tumbles and emit the results to another bolt
> where you update the count in Redis. Most of your state saving could also
> be automated by defining a Stateful bolt that would periodically checkpoint
> your state (sum per device). You could also club both windowing and state
> into a StatefulWindowedBolt implementation. You can evaluate the options
> and decide based on your use cases.
>
> Take a look at the sample topologies (SlidingWindowTopology,
> SlidingTupleTsTopology, StatefulTopology, StatefulWindowingTopology) in
> storm-starter and the docs for more info.
>
> https://github.com/apache/storm/blob/master/docs/Windowing.md
>
> https://github.com/apache/storm/blob/master/docs/State-checkpointing.md
>
>
> -Arun
>
>
>
>
> On 4/10/16, 4:30 PM, "Matthias J. Sax" <mj...@apache.org> wrote:
>
> >A tumbling window (ie, non-overlapping window) is the right approach (a
> >sliding window is overlapping).
> >
> >The window goes into your aggregation bolt (windowing and aggregation
> >goes hand in hand, ie, when the window gets closed, the aggregation is
> >triggered and the window content is handed over to the aggregation
> >function).
> >
> >Be aware that Storm (currently) only supports processing time window
> >(an no event time windows).
> >
> >-Matthias
> >
> >
> >On 04/10/2016 09:56 AM, Daniela Stoiber wrote:
> >> Hi,
> >>
> >> thank you for your reply.
> >>
> >> How can I ensure that the latest values are pulled from Redis the sum
> >> is updated every minute? Do I need a sliding window with an interval
> >> of 1 minute? Where would this sliding window be located in my topology?
> >>
> >> Thank you in advance.
> >>
> >> Regards,
> >> Daniela
> >>
> >> -----Ursprüngliche Nachricht-----
> >> Von: Matthias J. Sax [mailto:mj...@apache.org]
> >> Gesendet: Samstag, 9. April 2016 12:13
> >> An: dev@storm.apache.org
> >> Betreff: Re: Use only latest values
> >>
> >> Sounds reasonable.
> >>
> >>
> >> On 04/09/2016 08:34 AM, Daniela Stoiber wrote:
> >>> Hi,
> >>>
> >>>
> >>>
> >>> I would like to cache values and to use only the latest "valid"
> >>> values to build a sum.
> >>>
> >>> In more detail, I receive values from devices periodically. I would
> >>> like to add up all the valid values each minute. But not every
> >>> device sends a new value every minute. And as long as there is no
> >>> new value the old one should be used for the sum. As soon as I
> >>> receive a new value from a device I would like to overwrite the old
> >>> value and to use the new one for the sum. Would that be possible
> >>> with the combination of
> >> Storm and Redis?
> >>>
> >>>
> >>>
> >>> My idea was to use the following:
> >>>
> >>>
> >>>
> >>> - Kafka Spout
> >>>
> >>> - Storm Bolt for storing the tuples in Redis and for overwriting the
> >>> values as soon as a new one is delivered
> >>>
> >>> - Storm Bolt for reading the latest tuples from Redis
> >>>
> >>> - Storm Bolt for grouping (I would like to group the devices per
> >>> region)
> >>>
> >>> - Storm Bolt for aggregation
> >>>
> >>> - Storm Bolt for storing the results again in Redis
> >>>
> >>>
> >>>
> >>> Thank you in advance.
> >>>
> >>>
> >>>
> >>> Regards,
> >>>
> >>> Daniela
> >>>
> >>>
> >>
> >>
> >
>
>
>

Reply via email to