morningman opened a new pull request, #32340:
URL: https://github.com/apache/doris/pull/32340
## Proposed changes
File meta cache on BE is used to cache the meta for external table's file
such as parquet footer.
This cache is counted by number, not memory consumption.
So if the cache object is big(eg, a large parquet footer), the total memory
consumption of this cache
will be large and causing OOM.
This PR mainly changes:
1. Add a new method `exceed_prune_limit()` for `CachePolicy`
For `ObjLRUCache`, it always return true so that the minor of full gc on
BE will prune the cache each time.
2. Reduce the default capability of file meta cache, from 20000 to 1000
3. Change judgement of whether enable file meta cache when querying
If the number of file need to be read is larger than the 1/3 of the file
meta cache's capability, file meta cache
will be disabled for this query. Because cache is useless if there are
too many files.
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[[email protected]](mailto:[email protected]) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
