Nice find Gabe. That would certainly explain a lot. Turns out there's a 50Gbps quota that will return a 429 error if hit: https://cloud.google.com/storage/quotas. I'll see if I can get this increased and, either way, I'll implement a wait-and-retry strategy in the downloader for when this error is received. I suspect this is hit when lots of Kokoro instances are spun up at one time.
-- Dr. Bobby R. Bruce Room 3050, Kemper Hall, UC Davis Davis, CA, 95616 web: https://www.bobbybruce.net On Tue, Mar 1, 2022 at 2:08 AM Giacomo Travaglini via gem5-dev < gem5-dev@gem5.org> wrote: > Hi Gabe, > > > > A possible workaround could be to use the --bin-path option in testlib > > > > Kind Regards > > > > Giacomo > > > > *From: *Gabe Black via gem5-dev <gem5-dev@gem5.org> > *Date: *Tuesday, 1 March 2022 at 04:10 > *To: *gem5 Developer List <gem5-dev@gem5.org> > *Cc: *Gabe Black <gabe.bl...@gmail.com> > *Subject: *[gem5-dev] resource downloader failure > > Hi folks. I've been trying to run tests locally, and I've been running > into occasional flakiness due to a problem with the resource downloader. > This reminds me somewhat of flakiness I was seeing in kokoro as well, so > they are probably related. The error output is this: > > > > $ cat > ./testing-results/SuiteUID:atomic-cpu_1-cores_classic_kvm-fork-run-test-GCN3_X86-x86_64-opt/TestUID:atomic-cpu_1-cores_classic_kvm-fork-run-test-GCN3_X86-x86_64-opt/simerr > warn: <orphan X86Board>.kvm_vm already has parent not resetting parent. > Note: kvm_vm is not a parameter of X86Board > warn: (Previously declared as <orphan X86Board>.processor.kvm_vm > HTTPError: HTTP Error 429: Too Many Requests > > At: > /usr/lib/python3.10/urllib/request.py(643): http_error_default > /usr/lib/python3.10/urllib/request.py(496): _call_chain > /usr/lib/python3.10/urllib/request.py(563): error > /usr/lib/python3.10/urllib/request.py(634): http_response > /usr/lib/python3.10/urllib/request.py(525): open > /usr/lib/python3.10/urllib/request.py(216): urlopen > build/GCN3_X86/python/gem5/resources/downloader.py(77): > _get_resources_json > build/GCN3_X86/python/gem5/resources/downloader.py(207): > get_resources_json_obj > build/GCN3_X86/python/gem5/resources/resource.py(162): __init__ > /home/gblack/gem5/work/tests/gem5/configs/boot_kvm_fork_run.py(205): > <module> > build/GCN3_X86/python/m5/main.py(434): main > > > > > > Is there a limit that needs to be bumped up? Some way to consolidate > requests so we're not hitting the existing limit? > > > > Gabe > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > _______________________________________________ > gem5-dev mailing list -- gem5-dev@gem5.org > To unsubscribe send an email to gem5-dev-le...@gem5.org > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
_______________________________________________ gem5-dev mailing list -- gem5-dev@gem5.org To unsubscribe send an email to gem5-dev-le...@gem5.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
