Re: Clarifying aspects of write ahead logs
Hi Raymond, No archiver mode means there is no cyclic enumeration of files in the work directory, enumeration of segments is plain and absolute instead. And all walHistSize rules applied to default mode applies to no-archiver mode, as well. So there is not possible to lose any data, and no additional checkpoints will be forced. Sincerely, Dmitriy Pavlov ср, 3 окт. 2018 г. в 23:49, Raymond Wilson : > HI Dmitriy, > > > > Thanks for clarifying. > > > > If archiving is turned off, what happens when all the WAL segments are > filled before a checkpoint occurs? Is a checkpoint forced at that point, or > does Ignite increase the number of segment files used above workSegments, > or is there a potential for data loss? > > > > Raymond. > > > > *From:* Dmitriy Pavlov > *Sent:* Tuesday, October 2, 2018 8:42 AM > *To:* user@ignite.apache.org > *Subject:* Re: Clarifying aspects of write ahead logs > > > > Hi Raymond, > > > > WAL Archive can be used for recovery during node restart. It can happen if > WAL data during one checkpoint is longer than segmentSize*workSegments > (640Mb by default). > > > > The downside of not having archiver is following: Space allocation for > a file may require significant time in some file systems. If we archive one > WAL segment (asynchronously) we can later reuse this segment file in the > work directory for a new segment records. The file is now preallocated, > there will be no performance penalty for step by step increasing of its > volume. > > > > Sincerely, > > Dmitriy Pavlov > > > > You can find scheme of WAL rotation here: > https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-WALstructure > > > > > пн, 1 окт. 2018 г. в 22:21, Raymond Wilson : > > Hi, > > > > I’ve been reading through this page ( > https://apacheignite.readme.io/docs/write-ahead-log) on WAL files in > Ignite. > > > > I’d like to understand what the purpose of the WAL archive is. I > understand from the documentation that the WAL archive contains data that > has been checkpointed, and so safely committed to the cache partition files. > > > > It seems to be optional in that it is possible to not have one by setting > the WAL active and archive paths to be the same location. The documentation > recommends doing this if ingest volumes mean time spent copying WAL files > to the archive can cause node freezes. > > > > Is the WAL archive used in recovery operations when an Ignite node > restarts? Is there a downside to not having WAL segments archived? > > > > Thanks, > > Raymond. > > > >
RE: Clarifying aspects of write ahead logs
HI Dmitriy, Thanks for clarifying. If archiving is turned off, what happens when all the WAL segments are filled before a checkpoint occurs? Is a checkpoint forced at that point, or does Ignite increase the number of segment files used above workSegments, or is there a potential for data loss? Raymond. *From:* Dmitriy Pavlov *Sent:* Tuesday, October 2, 2018 8:42 AM *To:* user@ignite.apache.org *Subject:* Re: Clarifying aspects of write ahead logs Hi Raymond, WAL Archive can be used for recovery during node restart. It can happen if WAL data during one checkpoint is longer than segmentSize*workSegments (640Mb by default). The downside of not having archiver is following: Space allocation for a file may require significant time in some file systems. If we archive one WAL segment (asynchronously) we can later reuse this segment file in the work directory for a new segment records. The file is now preallocated, there will be no performance penalty for step by step increasing of its volume. Sincerely, Dmitriy Pavlov You can find scheme of WAL rotation here: https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-WALstructure пн, 1 окт. 2018 г. в 22:21, Raymond Wilson : Hi, I’ve been reading through this page ( https://apacheignite.readme.io/docs/write-ahead-log) on WAL files in Ignite. I’d like to understand what the purpose of the WAL archive is. I understand from the documentation that the WAL archive contains data that has been checkpointed, and so safely committed to the cache partition files. It seems to be optional in that it is possible to not have one by setting the WAL active and archive paths to be the same location. The documentation recommends doing this if ingest volumes mean time spent copying WAL files to the archive can cause node freezes. Is the WAL archive used in recovery operations when an Ignite node restarts? Is there a downside to not having WAL segments archived? Thanks, Raymond.
Re: Clarifying aspects of write ahead logs
Hi Raymond, WAL Archive can be used for recovery during node restart. It can happen if WAL data during one checkpoint is longer than segmentSize*workSegments (640Mb by default). The downside of not having archiver is following: Space allocation for a file may require significant time in some file systems. If we archive one WAL segment (asynchronously) we can later reuse this segment file in the work directory for a new segment records. The file is now preallocated, there will be no performance penalty for step by step increasing of its volume. Sincerely, Dmitriy Pavlov You can find scheme of WAL rotation here: https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-WALstructure пн, 1 окт. 2018 г. в 22:21, Raymond Wilson : > Hi, > > > > I’ve been reading through this page ( > https://apacheignite.readme.io/docs/write-ahead-log) on WAL files in > Ignite. > > > > I’d like to understand what the purpose of the WAL archive is. I > understand from the documentation that the WAL archive contains data that > has been checkpointed, and so safely committed to the cache partition files. > > > > It seems to be optional in that it is possible to not have one by setting > the WAL active and archive paths to be the same location. The documentation > recommends doing this if ingest volumes mean time spent copying WAL files > to the archive can cause node freezes. > > > > Is the WAL archive used in recovery operations when an Ignite node > restarts? Is there a downside to not having WAL segments archived? > > > > Thanks, > > Raymond. > > >
Clarifying aspects of write ahead logs
Hi, I’ve been reading through this page ( https://apacheignite.readme.io/docs/write-ahead-log) on WAL files in Ignite. I’d like to understand what the purpose of the WAL archive is. I understand from the documentation that the WAL archive contains data that has been checkpointed, and so safely committed to the cache partition files. It seems to be optional in that it is possible to not have one by setting the WAL active and archive paths to be the same location. The documentation recommends doing this if ingest volumes mean time spent copying WAL files to the archive can cause node freezes. Is the WAL archive used in recovery operations when an Ignite node restarts? Is there a downside to not having WAL segments archived? Thanks, Raymond.