hartmannathan commented on PR #18576: URL: https://github.com/apache/nuttx/pull/18576#issuecomment-4254405879
> @hartmannathan Awesome ideas! I was pondering your ideas with @simbit18, we hit a couple of roadblocks: Thanks @lupyuen @simbit18 ! I will try to answer: > > > > 1. How will we auto-refresh the Cached Dependencies in nuttx-ci-deps? To make sure they are always up-to-date? Who will watch over nuttx-ci-deps, and fix it if any downloads fail? > Interesting, I hadn't considered auto-refresh. My thoughts were that the nuttx-ci-deps repo would be populated with specific versions of the packages, that we would consider "blessed" versions for CI testing. When a NuttX release happens, we could document which versions of dependencies have been used in testing. Developers who wish to use more bleeding edge versions would of course have the option to do that. Perhaps after each release, we could evaluate the versions of dependencies that are available and decide whether to update the nuttx-ci-deps repo. This means there is a human in the loop who can verify that package is legit and so on. > > > 2. ESP32 Downloads might be tricky, I don't see any obvious way to hook the ESP32 Builds to our Download Script. Also: I think ESP32 Downloads might need to be refreshed more often. I'm not familiar with ESP32 so I'm unsure how to answer this. Could you tell me a bit more about how it gets downloaded? > > > > 3. Nearly all Download Failures are for URLs hosted at github.com. So we don't have much problems with downloads from External Third Parties. I suspect github.com is blocking our downloading because we run too many CI Jobs in parallel, all spamming github.com at the same time. (Hence I implemented Random Exponential Backoff) > In this case, maybe there should be a short (less than 10 seconds maybe) random delay before each download starts? That might cause the requests to be more serialized and perhaps avoid or reduce the failures. > > > 4. FYI chromium.googlesource.com was blocking our download of libyuv library, so we cached it at the NuttX Mirror Repo. libyuv hasn't been updated since 2021, so we probably won't need to worry about refreshing: > > - https://github.com/apache/nuttx-apps/pull/3449 This was exactly my thought, but for all downloaded deps. Our cached libyuv lib could move to a nuttx-ci-deps repo if we go this route. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
