Yes, it seems like keeping compactions around instead of deleting them would be useful. Even better if their usage is automatic when the consumer sets the 'read compacted' flag. Though a manual retrieval of the specific compaction shouldn't be too difficult.
-Patrick On Tue, May 5, 2020, at 14:01, Sijie Guo wrote: > Hi Patrick, > > Currently Pulsar only supports one compaction state per topic at this moment. > However the underneath implementation doesn't prevent supporting multiple > concurrent compactions as we just need to track the complications in > different cursors. As I understand here, you want the ability to keep the > compaction state as a snapshot hence you can go back in time to lookup the > compaction data. Is that the fair statement? > > - sijie > > On Mon, May 4, 2020 at 5:20 PM Patrick Hemmer <[email protected]> wrote: >> __ >> I'm looking to use Pulsar to store some timeseries data with infinite >> retention. For a simplistic example, lets say it's a bunch of orders >> lifecycle events (created, shipped, received, cancelled, etc). I want to be >> able to retrieve the state of the open orders at a given point in time, and >> then all order events up to a second point in time. >> >> At first compaction seems like what I want here, as with the first point in >> time, I don't care about orders in the past which are closed. So compaction >> would keep me from having to replay all events from the beginning of time. >> However as I want to be able to retrieve any period of time, I would need >> multiple compaction points so I could use the one closest to my start time. >> >> Is this possible? >> From my understanding reading through the documentation >> (https://pulsar.apache.org/docs/en/concepts-topic-compaction/), it looks >> like there's only a single compaction point. So if I performed a compaction >> today, but I wanted to retrieve last month's data, I'd have to replay from >> the beginning of time. >> Can Pulsar handle this, or will I have to create some manual method of >> snapshotting and storing the state at periodic intervals? >> >> Thanks >> >> -Patrick
