Our first big test run we wanted to eliminate as many variables as possible, so this is on 1 machine with 1 task manager and 1 parallelism. The machine has 4 disks though, and as you can see, they mostly all use around the same space for storage until a savepoint is triggered.
Could it be that given a parallelism of 1, certain operator's states are pinned to specific drives and as it's doing compaction it's moving everything over to that drive into a single file? In which case, would greater parallelism distribute the work more evenly? Thanks! On Sat, Dec 12, 2020 at 2:35 AM David Anderson <dander...@apache.org> wrote: > RocksDB does do compaction in the background, and incremental checkpoints > simply mirror to S3 the set of RocksDB SST files needed by the current set > of checkpoints. > > However, unlike checkpoints, which can be incremental, savepoints are > always full snapshots. As for why one host would have much more state than > the others, perhaps you have significant key skew, and one task manager is > ending up with more than its share of state to manage. > > Best, > David > > On Sat, Dec 12, 2020 at 12:31 AM Rex Fenley <r...@remind101.com> wrote: > >> Hi, >> >> We're using the Rocks state backend with incremental checkpoints and >> savepoints setup for S3. We notice that every time we trigger a savepoint, >> one of the local disks on our host explodes in disk usage. >> What is it that savepoints are doing which would cause so much disk to be >> used? >> Our checkpoints are a few GiB in size, is the savepoint combining all the >> checkpoints together at once on disk? >> I figured that incremental checkpoints would compact over time in the >> background, is that correct? >> >> Thanks >> >> Graph here. Parallelism is 1 and volume size is 256 GiB. >> [image: Screen Shot 2020-12-11 at 2.59.59 PM.png] >> >> >> -- >> >> Rex Fenley | Software Engineer - Mobile and Backend >> >> >> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> >> | FOLLOW US <https://twitter.com/remindhq> | LIKE US >> <https://www.facebook.com/remindhq> >> > -- Rex Fenley | Software Engineer - Mobile and Backend Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> | FOLLOW US <https://twitter.com/remindhq> | LIKE US <https://www.facebook.com/remindhq>
