> This keeps the performance stable, but I'm not sure how to persist the > expired data properly Perhaps you could listen to the EXPIRED events of the cache that stores the data you want to persist, create an additional cache with persistence enabled and put all your expired data there. Thus you’ll have your current caches with persistence disabled (there are two of them, right?) storing relevant data that you execute queries for, and one more cache with persistence enabled that you’ll only write old data to. Haven’t tried that myself though.
Stan From: svonn Sent: 22 января 2018 г. 20:34 To: user@ignite.apache.org Subject: Question about persisting stream processing results Hi! I'm receiving two streams of events, stream one (1) is basically only used as basis for interpolating and putting data in stream two (2). Whenever an element in stream (1) arrives, the local listener of my ContinuousQuery starting searching the previous element belonging to the same group. More specifically, its a ScanQuery that compares some IDs and searches for the one that has a timestamp bigger than the current one minus 1500ms while being smaller than the current timestamp. Currently, I want to persist stream (2) while keeping a stable performance. Whats the best way to do that? Simply activating Ignite persistence sounds like it's simply starting to move data from off-heap RAM to the harddrive when the RAM space is shrinking. However, if I understood it correctly, it will still query those elements for all my stream processing queries. So trying to find the previous element of stream (1) or trying to find all elements that are between those two elements in stream (2) would become slower and slower the longer the task runs. The incoming data is relevant for about 5 minutes, thus I tried using an expiration policy. This keeps the performance stable, but I'm not sure how to persist the expired data properly. Also, for calibration purposes, I'm generating a Map to store and apply calibration on elements - when I'm activating the expiry policy, I'm starting to run in Nullpointer Exceptions after about 5 minutes - is the policy also deleting the Map? Best regards Svonn -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/