Hi, I didn't get the point why it may be required to separate WAL work, WAL uncheckpointed archive (some work outside segment rotation) and checkpointed archive (which is better to be compressed using Ignite new feature - WAL compressor).
Please consider new no-archiver mode implemented recently. If archive folder confuses end user, grid admin may set up this mode (all segments is placed in 1 directory) instead of introducing folders. вт, 13 февр. 2018 г. в 22:11, Ivan Rakov <ivan.glu...@gmail.com>: > I think, I got the point now. > There's no need to copy files from "temp" to "archive" dir - we can just > move them, which is a constant-time operation. > Makes sense. > > Change is quite complex (we need to synchronize all movings thoroughly > to avoid ruining existing WAL read iterators), but feasible. > > Best Regards, > Ivan Rakov > > > On 13.02.2018 22:06, Ivan Rakov wrote: > > Yakov, > > > > This will work. However, I expect performance degradation with this > > change. Disk storage has a limited number of I/O operations per second > > on hardware level. List of already existing disk I/O activities > > (writing to WAL work dir, copying from WAL work dir to WAL archive > > dir, writing partition files during checkpoint) will be updated with a > > new one - copying from WAL work dir to temp dir. > > > > Best Regards, > > Ivan Rakov > > > > On 13.02.2018 21:35, Yakov Zhdanov wrote: > >> Ivan, > >> > >> I do not want to create new files. As far as I know, now we copy > >> segments > >> to archive dir before they get checkpointed. What I suggest is to > >> copy them > >> to a temp dir under wal directory and then move to archive. In my > >> understanding at the time we copy the files to a temp folder all > >> changes to > >> them are already fsynced. > >> > >> Correct? > >> > >> Yakov Zhdanov, > >> www.gridgain.com > >> > >> 2018-02-13 21:29 GMT+03:00 Ivan Rakov <ivan.glu...@gmail.com>: > >> > >>> Yakov, > >>> > >>> I see the only one problem with your suggestion - number of > >>> "uncheckpointed" segments is potentially unlimited. > >>> Right now we have limited number (10) of file segments with immutable > >>> names in WAL "work" directory. We have to keep this approach due to > >>> known > >>> bug in XFS - fsync time is nearly twice bigger for recently created > >>> files. > >>> > >>> Best Regards, > >>> Ivan Rakov > >>> > >>> > >>> On 13.02.2018 21:22, Yakov Zhdanov wrote: > >>> > >>>> I meant we still will be copying segment once and then will be > >>>> moving it > >>>> to > >>>> archive which should not affect file system much. > >>>> > >>>> Thoughts? > >>>> > >>>> --Yakov > >>>> > >>>> 2018-02-13 21:19 GMT+03:00 Yakov Zhdanov <yzhda...@apache.org>: > >>>> > >>>> Alex, > >>>>> I remember we had some confusing behavior for WAL archive when > >>>>> archived > >>>>> segments were required for successful recovery. > >>>>> > >>>>> Is issue still present? > >>>>> > >>>>> If yes, what if we copy "uncheckpointed" segments to a directory > >>>>> under > >>>>> wal > >>>>> directory and then move the segments to archive after checkpoint? > >>>>> Will > >>>>> this > >>>>> work? > >>>>> > >>>>> Thanks! > >>>>> > >>>>> --Yakov > >>>>> > >>>>> > > > >
