Re: Preventing field data from being loaded into page cache

2023-10-21 Thread Marcus Eagan
Justin, Be advised, overdependence on DirectIODirectory can lead to other problems, which vary based on the amount of data, access patterns, resource utilization, etc. I have seen issues in a few places. If you are running a fork, the problems can be even more pronounced and less understood. You w

Re: Preventing field data from being loaded into page cache

2023-10-21 Thread Uwe Schindler
Hi, There is a workaround available called DirectIODirectory. You can subclass it and override useDirectIO() method to return true only for fdt files. It wraps another FSDirectory (e.g. MMapDirectory) and delegates everything back to it, but for those where useDirectIO() returns true it imple

Preventing field data from being loaded into page cache

2023-10-20 Thread Justin Borromeo
Is there any way to keep field data files out of the operating system's page cache? We only use fdt for highlighting and don't need to keep it warm in memory. From what I understand, the operating system is in control of what files get loaded into the page cache. Does Lucene have any mechanisms to