Hi Daniela, Then have keys like this for every entry YYYYMMDDHHmm_DeviceId and LATEST_DeviceId
So whenever a new value comes set both of above keys. Latest will always get ovveride and Minute one may or may not get override depending on if you get the same data with in a minute That way you will keep data in minute window as well as latest one will be available in latest key. I assume you will not set any expiry time for latest key but may wanna set some expiry time for YYYYMMDDHHmm_DeviceId Thanks Ravi. On Mon, Apr 11, 2016 at 1:02 PM, Daniela Stoiber <daniela.stoi...@gmail.com> wrote: > Hi Ravi > > Thank you for your reply. > > Unfortunately I cannot set the expiry time to 2 minutes as I need the value > as long as there is no new value. So I need one value for each device every > minute and always the most recent value for each device. > > Regards, > Daniela > > > 2016-04-11 13:16 GMT+02:00 Ravi Sharma <ping2r...@gmail.com>: > > > Hi Daniel, > > > > What about creating your Redis Key in format YYYYMMDDHHmm_DeviceId or may > > be just HHmm_DeviceId > > > > i.e. > > 201604111213_A or 1213_A > > > > and give the key expiry time as 2 minutes. i assume you dont care about > > past data. > > > > Now whenevr a new value come set this key Value (YYYYMMDDHHmm_DeviceId or > > HHmm_DeviceId) > > > > and on aggregation side whenever you have to add all values, just get the > > list of all of ur devices, build all keys with ur format and do a > multiget > > for all keys i.e 201604111213_A, 201604111213_B etc and add it. I think > > redis also has a way to do the sum and save the result into other key. > > > > > > Thanks > > Ravi. > > > > > > > > On Sun, Apr 10, 2016 at 9:46 PM, Daniela Stoiber < > > daniela.stoi...@gmail.com> > > wrote: > > > > > HI Arun, > > > > > > thank you for your reply. > > > > > > But my problem is that I need to add up the values over all devices, > but > > I > > > am only allowed to use the most recent value of each device. A value is > > > valid as long as there is no new value for this device available. > > > > > > So if I receive a message with device A with value 1, value 1 should be > > > used for the sum as long as the value of A does not change. > > > When I receive a new value for A, the new value should be used for the > > sum > > > and the old one should be replaced. > > > > > > Therefore I thought to use Redis to store this information: > > > Device Value > > > A 1 > > > B 10 > > > C 4 > > > > > > Then I would like to pull every minute the most recent value of each > > > device to build the sum. Therefore I would like to use the windowed > bolt. > > > But I am not sure if it is possible to pull data out of Redis within a > > > windowed bolt. > > > > > > Thank you in advance. > > > > > > Regards, > > > Daniela > > > > > > > > > -----Ursprüngliche Nachricht----- > > > Von: Arun Iyer [mailto:ai...@hortonworks.com] Im Auftrag von Arun > > > Mahadevan > > > Gesendet: Sonntag, 10. April 2016 20:55 > > > An: dev@storm.apache.org > > > Betreff: Re: AW: Use only latest values > > > > > > Hi Matthias, > > > > > > WindowedBolt does support event time. In trident its is not yet > exposed. > > > > > > Hi Daniela, > > > > > > You could solve your use cases in different ways. One would be to have > a > > > WindowedBolt with a 1 min tumbling window, do your custom aggregation > > (e.g. > > > sum) every time the window tumbles and emit the results to another bolt > > > where you update the count in Redis. Most of your state saving could > also > > > be automated by defining a Stateful bolt that would periodically > > checkpoint > > > your state (sum per device). You could also club both windowing and > state > > > into a StatefulWindowedBolt implementation. You can evaluate the > options > > > and decide based on your use cases. > > > > > > Take a look at the sample topologies (SlidingWindowTopology, > > > SlidingTupleTsTopology, StatefulTopology, StatefulWindowingTopology) in > > > storm-starter and the docs for more info. > > > > > > https://github.com/apache/storm/blob/master/docs/Windowing.md > > > > > > > https://github.com/apache/storm/blob/master/docs/State-checkpointing.md > > > > > > > > > -Arun > > > > > > > > > > > > > > > On 4/10/16, 4:30 PM, "Matthias J. Sax" <mj...@apache.org> wrote: > > > > > > >A tumbling window (ie, non-overlapping window) is the right approach > (a > > > >sliding window is overlapping). > > > > > > > >The window goes into your aggregation bolt (windowing and aggregation > > > >goes hand in hand, ie, when the window gets closed, the aggregation is > > > >triggered and the window content is handed over to the aggregation > > > >function). > > > > > > > >Be aware that Storm (currently) only supports processing time window > > > >(an no event time windows). > > > > > > > >-Matthias > > > > > > > > > > > >On 04/10/2016 09:56 AM, Daniela Stoiber wrote: > > > >> Hi, > > > >> > > > >> thank you for your reply. > > > >> > > > >> How can I ensure that the latest values are pulled from Redis the > sum > > > >> is updated every minute? Do I need a sliding window with an interval > > > >> of 1 minute? Where would this sliding window be located in my > > topology? > > > >> > > > >> Thank you in advance. > > > >> > > > >> Regards, > > > >> Daniela > > > >> > > > >> -----Ursprüngliche Nachricht----- > > > >> Von: Matthias J. Sax [mailto:mj...@apache.org] > > > >> Gesendet: Samstag, 9. April 2016 12:13 > > > >> An: dev@storm.apache.org > > > >> Betreff: Re: Use only latest values > > > >> > > > >> Sounds reasonable. > > > >> > > > >> > > > >> On 04/09/2016 08:34 AM, Daniela Stoiber wrote: > > > >>> Hi, > > > >>> > > > >>> > > > >>> > > > >>> I would like to cache values and to use only the latest "valid" > > > >>> values to build a sum. > > > >>> > > > >>> In more detail, I receive values from devices periodically. I would > > > >>> like to add up all the valid values each minute. But not every > > > >>> device sends a new value every minute. And as long as there is no > > > >>> new value the old one should be used for the sum. As soon as I > > > >>> receive a new value from a device I would like to overwrite the old > > > >>> value and to use the new one for the sum. Would that be possible > > > >>> with the combination of > > > >> Storm and Redis? > > > >>> > > > >>> > > > >>> > > > >>> My idea was to use the following: > > > >>> > > > >>> > > > >>> > > > >>> - Kafka Spout > > > >>> > > > >>> - Storm Bolt for storing the tuples in Redis and for overwriting > the > > > >>> values as soon as a new one is delivered > > > >>> > > > >>> - Storm Bolt for reading the latest tuples from Redis > > > >>> > > > >>> - Storm Bolt for grouping (I would like to group the devices per > > > >>> region) > > > >>> > > > >>> - Storm Bolt for aggregation > > > >>> > > > >>> - Storm Bolt for storing the results again in Redis > > > >>> > > > >>> > > > >>> > > > >>> Thank you in advance. > > > >>> > > > >>> > > > >>> > > > >>> Regards, > > > >>> > > > >>> Daniela > > > >>> > > > >>> > > > >> > > > >> > > > > > > > > > > > > > > > >
