Hi, On 2023/12/11 10:02, Yoann Congal wrote: >Hello, > >Le 11/12/2023 à 08:51, Yuta Hayama a écrit : >> Hi, >> >> On 2023/12/08 14:04, Dhairya Nagodra via lists.openembedded.org wrote: >>> Sometimes NVD servers are unstable and return too many errors. >>> There is an option to have higher fetch attempts to increase the >>> chances of successfully fetching the CVE data. >>> >>> Additionally, it also makes sense to progressively increase the delay >>> after a failed request to an already unstable or busy server. >>> The increase in delay is reset after every successful request and the >>> maximum delay is limited to 30 seconds. >>> >>> Also, the logs are improved to give more clarity. >>> >>> Signed-off-by: Dhairya Nagodra <dnago...@cisco.com> >> >> I was just working on a similar issue. >> As a specific example, multiple cve-update-nvd2-native:do_fetch runs >> in parallel can easily reach the rate limit. It can be assumed that >> this situation will occur if several people run bitbake in one office. >> (often unaware of each >> other...) >> >> I have observed the do_fetch logs and found that HTTP 403 errors are >> returned if the request is blocked, probably due to rate limitation.
As per my knowledge, HTTP 403 is related to a permission issue rather than a rate limitation. I have not seen an HTTP 403 error, anytime. Can you please help clarify how was it generated? Is it reproducible? I tried removing both sleep delays altogether (and without API keys) to try and generate an error. In that, I couldn't generate any errors instead, got the response with a 15-20 sec delay. So, I guess it stayed within the rate limit. > >Shouldn't we ask the NVD to return "429 Too Many Requests" instead? >https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429 > >> NOTE: Requesting >> https://services.nvd.nist.gov/rest/json/cves/2.0?startIndex=6000 >> NOTE: CVE database: received error (HTTP Error 403: Forbidden), >> retrying >> NOTE: CVE database: received error (HTTP Error 403: Forbidden), >> retrying >> NOTE: CVE database: received error (HTTP Error 403: Forbidden), >> retrying >> NOTE: CVE database: received error (HTTP Error 403: Forbidden), >> retrying >> NOTE: CVE database: received error (HTTP Error 403: Forbidden), >> retrying >> WARNING: CVE database update failed >> DEBUG: Python function do_fetch finished >> >> Other times a request may fail with IncompleteRead, but this is >> clearly distinguishable from an HTTP error. >> >> Here, we can think of the following ideas. >> If an HTTP error occurs, assume that the rate limit has already been >> reached and wait 30 seconds to ensure that the next window starts. The >> patch will be something like this. >> >> --- >> meta/recipes-core/meta/cve-update-nvd2-native.bb | 5 ++++- >> 1 file changed, 4 insertions(+), 1 deletion(-) > >Regards, > >-- >Yoann Congal >Smile ECS - Tech Expert Regards, Dhairya
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#192157): https://lists.openembedded.org/g/openembedded-core/message/192157 Mute This Topic: https://lists.openembedded.org/mt/103049968/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
