Le jeudi 19 décembre 2013 04:41:18, Trent Piepho a écrit : > Do you see page file activity? If you look at /proc/pid/smaps, you > should be able to see the actual status of the mapping of your data > file. Probably it is consuming a large number of pages of RAM, but > also there should be zero pages written to swap. All clean private or > clean shared, zero anonymous and zero swap.
7f8e6f75d000-7f9334df1000 r--p 00000000 08:06 11645756 /home/even/Téléchargements/eudem_dem_4258_europe.tif Size: 20011600 kB Rss: 2943372 kB Pss: 2943372 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 2943372 kB Private_Dirty: 0 kB Referenced: 950596 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB So yes, you are right. I've noticed the 'Referenced' values tends to fluctuate a lot during execution, while other values remain stable after some point. > > I think the system unresponsiveness is probably do to I/O scheduling. > You're process has queued a lot of I/O reads and everything else has > to wait in the queue. So all other I/O sees huge latencies. Yes, that's a likely clause. I've just tried to do a madvise(..,..,MADV_RANDOM) on the whole mapping just after mmap(), and it seems to increase the system responsiveness in a noticeable way (since I can write this email while the program is running !). Of course, the throughput of the test program has reduced significantly (which is logical since agressive I/O read-ahead must be disabled). > > And too, a 20 GB mapping is probably thrashing the TLB. Do huge pages > actually get used? No, they don't. My understanding and previous attempts is that mmap() needs an explicit flag for that, and other tunings at OS level. > On the embedded systems I'm more intimately > familiar with, only normal 4k pages are used by user processes. Huge > TLBs are more of a special case that can be used by the kernel for > things like frame buffer mappings and SoC register windows. > -- Geospatial professional services http://even.rouault.free.fr/services.html _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev