Hi Daniel, What about creating your Redis Key in format YYYYMMDDHHmm_DeviceId or may be just HHmm_DeviceId
i.e. 201604111213_A or 1213_A and give the key expiry time as 2 minutes. i assume you dont care about past data. Now whenevr a new value come set this key Value (YYYYMMDDHHmm_DeviceId or HHmm_DeviceId) and on aggregation side whenever you have to add all values, just get the list of all of ur devices, build all keys with ur format and do a multiget for all keys i.e 201604111213_A, 201604111213_B etc and add it. I think redis also has a way to do the sum and save the result into other key. Thanks Ravi. On Sun, Apr 10, 2016 at 9:46 PM, Daniela Stoiber <daniela.stoi...@gmail.com> wrote: > HI Arun, > > thank you for your reply. > > But my problem is that I need to add up the values over all devices, but I > am only allowed to use the most recent value of each device. A value is > valid as long as there is no new value for this device available. > > So if I receive a message with device A with value 1, value 1 should be > used for the sum as long as the value of A does not change. > When I receive a new value for A, the new value should be used for the sum > and the old one should be replaced. > > Therefore I thought to use Redis to store this information: > Device Value > A 1 > B 10 > C 4 > > Then I would like to pull every minute the most recent value of each > device to build the sum. Therefore I would like to use the windowed bolt. > But I am not sure if it is possible to pull data out of Redis within a > windowed bolt. > > Thank you in advance. > > Regards, > Daniela > > > -----Ursprüngliche Nachricht----- > Von: Arun Iyer [mailto:ai...@hortonworks.com] Im Auftrag von Arun > Mahadevan > Gesendet: Sonntag, 10. April 2016 20:55 > An: dev@storm.apache.org > Betreff: Re: AW: Use only latest values > > Hi Matthias, > > WindowedBolt does support event time. In trident its is not yet exposed. > > Hi Daniela, > > You could solve your use cases in different ways. One would be to have a > WindowedBolt with a 1 min tumbling window, do your custom aggregation (e.g. > sum) every time the window tumbles and emit the results to another bolt > where you update the count in Redis. Most of your state saving could also > be automated by defining a Stateful bolt that would periodically checkpoint > your state (sum per device). You could also club both windowing and state > into a StatefulWindowedBolt implementation. You can evaluate the options > and decide based on your use cases. > > Take a look at the sample topologies (SlidingWindowTopology, > SlidingTupleTsTopology, StatefulTopology, StatefulWindowingTopology) in > storm-starter and the docs for more info. > > https://github.com/apache/storm/blob/master/docs/Windowing.md > > https://github.com/apache/storm/blob/master/docs/State-checkpointing.md > > > -Arun > > > > > On 4/10/16, 4:30 PM, "Matthias J. Sax" <mj...@apache.org> wrote: > > >A tumbling window (ie, non-overlapping window) is the right approach (a > >sliding window is overlapping). > > > >The window goes into your aggregation bolt (windowing and aggregation > >goes hand in hand, ie, when the window gets closed, the aggregation is > >triggered and the window content is handed over to the aggregation > >function). > > > >Be aware that Storm (currently) only supports processing time window > >(an no event time windows). > > > >-Matthias > > > > > >On 04/10/2016 09:56 AM, Daniela Stoiber wrote: > >> Hi, > >> > >> thank you for your reply. > >> > >> How can I ensure that the latest values are pulled from Redis the sum > >> is updated every minute? Do I need a sliding window with an interval > >> of 1 minute? Where would this sliding window be located in my topology? > >> > >> Thank you in advance. > >> > >> Regards, > >> Daniela > >> > >> -----Ursprüngliche Nachricht----- > >> Von: Matthias J. Sax [mailto:mj...@apache.org] > >> Gesendet: Samstag, 9. April 2016 12:13 > >> An: dev@storm.apache.org > >> Betreff: Re: Use only latest values > >> > >> Sounds reasonable. > >> > >> > >> On 04/09/2016 08:34 AM, Daniela Stoiber wrote: > >>> Hi, > >>> > >>> > >>> > >>> I would like to cache values and to use only the latest "valid" > >>> values to build a sum. > >>> > >>> In more detail, I receive values from devices periodically. I would > >>> like to add up all the valid values each minute. But not every > >>> device sends a new value every minute. And as long as there is no > >>> new value the old one should be used for the sum. As soon as I > >>> receive a new value from a device I would like to overwrite the old > >>> value and to use the new one for the sum. Would that be possible > >>> with the combination of > >> Storm and Redis? > >>> > >>> > >>> > >>> My idea was to use the following: > >>> > >>> > >>> > >>> - Kafka Spout > >>> > >>> - Storm Bolt for storing the tuples in Redis and for overwriting the > >>> values as soon as a new one is delivered > >>> > >>> - Storm Bolt for reading the latest tuples from Redis > >>> > >>> - Storm Bolt for grouping (I would like to group the devices per > >>> region) > >>> > >>> - Storm Bolt for aggregation > >>> > >>> - Storm Bolt for storing the results again in Redis > >>> > >>> > >>> > >>> Thank you in advance. > >>> > >>> > >>> > >>> Regards, > >>> > >>> Daniela > >>> > >>> > >> > >> > > > > >