Hi! I'm receiving two streams of events, stream one (1) is basically only used as basis for interpolating and putting data in stream two (2). Whenever an element in stream (1) arrives, the local listener of my ContinuousQuery starting searching the previous element belonging to the same group. More specifically, its a ScanQuery that compares some IDs and searches for the one that has a timestamp bigger than the current one minus 1500ms while being smaller than the current timestamp.
Currently, I want to persist stream (2) while keeping a stable performance. Whats the best way to do that? Simply activating Ignite persistence sounds like it's simply starting to move data from off-heap RAM to the harddrive when the RAM space is shrinking. However, if I understood it correctly, it will still query those elements for all my stream processing queries. So trying to find the previous element of stream (1) or trying to find all elements that are between those two elements in stream (2) would become slower and slower the longer the task runs. The incoming data is relevant for about 5 minutes, thus I tried using an expiration policy. This keeps the performance stable, but I'm not sure how to persist the expired data properly. Also, for calibration purposes, I'm generating a Map to store and apply calibration on elements - when I'm activating the expiry policy, I'm starting to run in Nullpointer Exceptions after about 5 minutes - is the policy also deleting the Map? Best regards Svonn -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
