[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469487#comment-16469487
]
ASF GitHub Bot commented on DRILL-5270:
---------------------------------------
ilooner commented on issue #1250: DRILL-5270: Improve loading of profiles
listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#issuecomment-387873929
@kkhatua
I think I understand the difference in our two perspectives. You wanted a
cache that will always only contain the **N** most recently created profiles.
If you happen to access the **N + 1**th youngest profile, the cache will not
contain it and will never contain it, the cache will only hold the **N** most
recently created profiles.
I still prefer the approach with the Guava cache because you can still
effectively achieve the same result. As new profiles are created they can be
added to the cache. If you access a very old profile, one more recently created
profile will be evicted from the cache and the old profile will be added to the
cache since a user just requested it. I would argue this behavior is not only
easier to implement since we are leveraging a library, but actually more
desirable since it caches a profile based on when it is used, not when it was
created.
If you still disagree with using the Guava cache. I agree with your proposal
of moving your cache into a separate class. I think you should also add some
unit tests for the cache to verify that it works as expected. The unit tests
will also make maintaining and enhancing the class easier for future developers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve loading of profiles listing in the WebUI
> ------------------------------------------------
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
> Affects Versions: 1.9.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Priority: Major
> Fix For: 1.14.0
>
>
> Currently, as the number of profiles increase, we reload the same list of
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32
> core server. With the caching, we can get it down to as much as a few
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the
> directory to confirm whether a reload is needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
