Bobby Bruce has uploaded this change for review. (
https://gem5-review.googlesource.com/c/public/gem5/+/57275 )
Change subject: stdlib: Cache the resources.json download
......................................................................
stdlib: Cache the resources.json download
"resources.json" is referenced when a resource is requested. The
"resources.json" file may be updated at any time and therefore, the
downloader was engineered to retrieve this file from the gem5-resources
repo on each request. However, this can lead to excessively frequently
pulls over short periods of time which is uncessary given how
infrequently this file changes.
To combat this, this patch caches the "resources.json" file and will
use it for up to an hour after creation before re-retrieving it.
Change-Id: I3b4907cbadce8a54df21d85f8021bf3603ae0f6f
---
M src/python/gem5/resources/downloader.py
1 file changed, 32 insertions(+), 2 deletions(-)
diff --git a/src/python/gem5/resources/downloader.py
b/src/python/gem5/resources/downloader.py
index 2647c56..db68b82 100644
--- a/src/python/gem5/resources/downloader.py
+++ b/src/python/gem5/resources/downloader.py
@@ -59,17 +59,23 @@
return uri
-def _get_resources_json_at_url(url: str) -> Dict:
+def _get_resources_json_at_url(url: str, use_caching: bool = True) -> Dict:
'''
Returns a resource JSON, in the form of a Python Dict. The URL location
of the JSON must be specified.
+ If `use_caching` is True, a copy of the JSON will be cached locally,
and
+ used for up to an hour after retrieval.
+
**Note**: The URL is assumed to be the location within a Google Source
repository. Special processing is done to handle this. This is the
primary
reason there are separate functions for handling the retrieving of the
resources JSON comapared to just using the `_download` function
directly.
:param url: The URL of the JSON file.
+ :param use_caching: True if a cached file is to be used (up to an
hour),
+ otherwise the file will be retrieved from the URL regardless. True by
+ default.
'''
file_path = os.path.join(
@@ -77,7 +83,12 @@
f"gem5-resources-{hashlib.md5(url.encode()).hexdigest()}.base64",
)
- _download(url, file_path)
+ # The resources.json file can change at any time, but to avoid
excessive
+ # retrieval we cache a version locally and use it for up to an hour
before
+ # obtaining a fresh copy.
+ if not use_caching or not os.path.exists(file_path) or \
+ (time.time() - os.path.getmtime(file_path)) > 3600:
+ _download(url, os.path.join(gettempdir(), file_path))
# Note: Google Source does not properly support obtaining files as raw
# text. Therefore when we open the URL we receive the JSON in base64
--
To view, visit https://gem5-review.googlesource.com/c/public/gem5/+/57275
To unsubscribe, or for help writing mail filters, visit
https://gem5-review.googlesource.com/settings
Gerrit-Project: public/gem5
Gerrit-Branch: develop
Gerrit-Change-Id: I3b4907cbadce8a54df21d85f8021bf3603ae0f6f
Gerrit-Change-Number: 57275
Gerrit-PatchSet: 1
Gerrit-Owner: Bobby Bruce <bbr...@ucdavis.edu>
Gerrit-MessageType: newchange
_______________________________________________
gem5-dev mailing list -- gem5-dev@gem5.org
To unsubscribe send an email to gem5-dev-le...@gem5.org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
