Bobby Bruce has submitted this change. (
https://gem5-review.googlesource.com/c/public/gem5/+/57275 )
(
3 is the latest approved patch-set.
No files were changed between the latest approved patch-set and the
submitted one.
)Change subject: stdlib: Cache the resources.json download
......................................................................
stdlib: Cache the resources.json download
"resources.json" is referenced when a resource is requested. The
"resources.json" file may be updated at any time and therefore, the
downloader was engineered to retrieve this file from the gem5-resources
repo on each request. However, this can lead to excessively frequently
pulls over short periods of time which is uncessary given how
infrequently this file changes.
To combat this, this patch caches the "resources.json" file and will
use it for up to an hour after creation before re-retrieving it.
Change-Id: I3b4907cbadce8a54df21d85f8021bf3603ae0f6f
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/57275
Reviewed-by: Gabe Black <gabe.bl...@gmail.com>
Maintainer: Jason Lowe-Power <power...@gmail.com>
Tested-by: kokoro <noreply+kok...@google.com>
---
M src/python/gem5/resources/downloader.py
1 file changed, 43 insertions(+), 2 deletions(-)
Approvals:
Gabe Black: Looks good to me, approved
Jason Lowe-Power: Looks good to me, approved
kokoro: Regressions pass
diff --git a/src/python/gem5/resources/downloader.py
b/src/python/gem5/resources/downloader.py
index 239860c..5ca7387 100644
--- a/src/python/gem5/resources/downloader.py
+++ b/src/python/gem5/resources/downloader.py
@@ -59,17 +59,23 @@
return uri
-def _get_resources_json_at_url(url: str) -> Dict:
+def _get_resources_json_at_url(url: str, use_caching: bool = True) -> Dict:
'''
Returns a resource JSON, in the form of a Python Dict. The URL location
of the JSON must be specified.
+ If `use_caching` is True, a copy of the JSON will be cached locally,
and
+ used for up to an hour after retrieval.
+
**Note**: The URL is assumed to be the location within a Google Source
repository. Special processing is done to handle this. This is the
primary
reason there are separate functions for handling the retrieving of the
resources JSON comapared to just using the `_download` function
directly.
:param url: The URL of the JSON file.
+ :param use_caching: True if a cached file is to be used (up to an
hour),
+ otherwise the file will be retrieved from the URL regardless. True by
+ default.
'''
file_path = os.path.join(
@@ -77,7 +83,19 @@
f"gem5-resources-{hashlib.md5(url.encode()).hexdigest()}.base64",
)
- _download(url, file_path)
+ # The resources.json file can change at any time, but to avoid
excessive
+ # retrieval we cache a version locally and use it for up to an hour
before
+ # obtaining a fresh copy.
+ #
+ # `time.time()` and `os.path.getmtime(..)` both return an unix epoch
time
+ # in seconds. Therefore, the value of "3600" here represents an hour
+ # difference between the two values. `time.time()` gets the current
time,
+ # and `os.path.getmtime(<file>)` gets the modification time of the
file.
+ # This is the most portable solution as other ideas, like "file
creation
+ # time", are not always the same concept between operating systems.
+ if not use_caching or not os.path.exists(file_path) or \
+ (time.time() - os.path.getmtime(file_path)) > 3600:
+ _download(url, file_path)
# Note: Google Source does not properly support obtaining files as raw
# text. Therefore when we open the URL we receive the JSON in base64
--
To view, visit https://gem5-review.googlesource.com/c/public/gem5/+/57275
To unsubscribe, or for help writing mail filters, visit
https://gem5-review.googlesource.com/settings
Gerrit-Project: public/gem5
Gerrit-Branch: develop
Gerrit-Change-Id: I3b4907cbadce8a54df21d85f8021bf3603ae0f6f
Gerrit-Change-Number: 57275
Gerrit-PatchSet: 5
Gerrit-Owner: Bobby Bruce <bbr...@ucdavis.edu>
Gerrit-Reviewer: Bobby Bruce <bbr...@ucdavis.edu>
Gerrit-Reviewer: Daniel Carvalho <oda...@yahoo.com.br>
Gerrit-Reviewer: Gabe Black <gabe.bl...@gmail.com>
Gerrit-Reviewer: Giacomo Travaglini <giacomo.travagl...@arm.com>
Gerrit-Reviewer: Jason Lowe-Power <ja...@lowepower.com>
Gerrit-Reviewer: Jason Lowe-Power <power...@gmail.com>
Gerrit-Reviewer: kokoro <noreply+kok...@google.com>
Gerrit-MessageType: merged
_______________________________________________
gem5-dev mailing list -- gem5-dev@gem5.org
To unsubscribe send an email to gem5-dev-le...@gem5.org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
