deniskuzZ commented on PR #4946: URL: https://github.com/apache/hive/pull/4946#issuecomment-1867748174
> > > @deniskuzZ However, although this solves the problem, I have another doubt. If we use static cache, then the elements in SplitGroup.cache[] will never be deleted, when the user changes the partition, the elements here are not updated, so is there a problem? For PRE-JOB this is not too much of a problem, but for LLAP it is clearly very problematic. So. really. maybe set cache is non-static is a not bad idea.At least this way, the data in the cache won't be leaked. > > > > > > Yes, I think the static cache will never be cleaned for LLAP, so this maybe lead some issue. But it seems this cache has been in there for long long time ([HIVE-8409](https://issues.apache.org/jira/browse/HIVE-8409) -> [HIVE-9976](https://issues.apache.org/jira/browse/HIVE-9976)), and i don't have deep knowledge about this code at present. I'd like to hear others opinions. > > @zhangbutao Hi. butao. I read the code for [HIVE-16079](https://issues.apache.org/jira/browse/HIVE-16079), and I think that we can use non-static caching without re-triggering the OOM problem. This is because [HIVE-16079](https://issues.apache.org/jira/browse/HIVE-16079) handles this by INTERNER all strings and properties objects. For example, CopyOnFirstWriteProperties::INTERNER. Since we already have a static cache based on the JVM, I don't think it's necessary to set SplitGroup.cache[] to be static, given the scenario in which LLAP works. > > @deniskuzZ However, although this solves the problem, I have another doubt. If we use static cache, then the elements in SplitGroup.cache[] will never be deleted, when the user changes the partition, the elements here are not updated, so is there a problem? For PRE-JOB this is not too much of a problem, but for LLAP it is clearly very problematic. So. really. maybe set cache is non-static is a not bad idea.At least this way, the data in the cache won't be leaked. > > Yes, I think the static cache will never be cleaned for LLAP, so this maybe lead some issue. But it seems this cache has been in there for long long time ([HIVE-8409](https://issues.apache.org/jira/browse/HIVE-8409) -> [HIVE-9976](https://issues.apache.org/jira/browse/HIVE-9976)), and i don't have deep knowledge about this code at present. I'd like to hear others opinions. Could we replace this map-based cache with caffeine cache? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
