>From our own experience with journaling the benefits of pre-touching files depends on the file system used. When ended up using XFS and threw away all of our pre-touching code as it had no benefit there. I would recommend testing with a wide variety of file systems.
Mike. On 11 July 2017 at 07:11, Vitaly Davidovich <vita...@gmail.com> wrote: > A few suggestions: > > 1) have you tried just reading the data in the prefaulting code, instead > of dirtying it with a dummy write? Since this is a disk backed mapping, it > should page fault and map the underlying file data (rather than mapping to > a zero page, e.g.). At a high rate of dirtying, this will amplify the > number of dirty pages and will add writeback pressure. Are you seeing > writeback during the tests by the way? How much disk io is happening? Are > the cores you're running on spending significant %iowait time? > > 2) have you tried playing with mmap and/or madvise hints? E.g. might be > interesting to prefault the mapping with MAP_POPULATE, turn off your manual > prefaulting, and see tail latency. Or madvise a sequential hint and see > what happens with kernels own readahead/prefault heuristics. Some of this > may not apply to OS X, I'm not that familiar with it. > > 3) as others have mentioned, try doing this benchmarking on Linux with > something like perf and see where the counters end up between the different > experiments. > > > On Mon, Jul 10, 2017 at 2:41 PM Roman Leventov <leventov...@gmail.com> > wrote: > >> On 10 July 2017 at 13:21, Martin Thompson <mjpt...@gmail.com> wrote: >> >>> Are you running recent MacOS with APFS which supports sparse files? >>> >> >> If this is of any interest, my OS version is >> >> Darwin MacBook-Pro.local 16.6.0 Darwin Kernel Version 16.6.0: Fri Apr 14 >> 16:21:16 PDT 2017; root:xnu-3789.60.24~6/RELEASE_X86_64 x86_64 >> >> (i. e. very recent) >> >> When considering an algorithm for pre-touch ahead it is often useful to >>> consider rate to determine who far ahead you should be pre-touching. >>> >> >> Logically the higher the rate the longer pre-touch chunk should be, but >> when I did pretouch of the whole 32 MB buffers (instead of 1 MB chunks) >> pauses were even worse. >> >> >>> When pre-touching then it can be better with a positional write of a >>> byte, i.e. use https://docs.oracle.com/javase/7/docs/api/java/nio/ >>> channels/FileChannel.html#write(java.nio.ByteBuffer,%20long) which >>> should map to pwrite() on Linux. This will be a safepoint whereas the >>> mapped buffer write will not be. >>> >> >> Thanks, will try >> >> -- >> You received this message because you are subscribed to the Google Groups >> "mechanical-sympathy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to mechanical-sympathy+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > -- > Sent from my phone > > -- > You received this message because you are subscribed to the Google Groups > "mechanical-sympathy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to mechanical-sympathy+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
