Just a quick status update: The artifact storage backend has been cut over, but a final sync task is still ongoing to sync the last delta of artifacts from S3 -> GCS. So if you see any task failures around missing artifacts, this is expected. Once the sync task wraps up in the next 30min or so, the artifacts should be available again. At which point rerunning your tasks should make them green.
On Wed, Jul 12, 2023 at 11:05 AM Andrew Halberstadt <[email protected]> wrote: > Quick reminder that the planned tree closure is scheduled for 12:00 EST > (one hour from now) and should last approximately two hours. Try will > remain open. Updates will be posted to #firefox-ci > <https://matrix.to/#/%23firefox-ci:mozilla.org> on Matrix and I'll > follow-up here when the migration is finished as well. > > Please be on the lookout for suspicious failures, especially around: > 1. Missing artifacts. > 2. Failure to decompress artifacts. > > Please ping the #firefox-ci > <https://matrix.to/#/%23firefox-ci:mozilla.org> channel on Matrix if you > find any issues. > Thanks, > Andrew > > On Fri, Jun 23, 2023 at 2:43 PM Andrew Halberstadt <[email protected]> > wrote: > >> *Update:* Firefox CI artifact migration and planned tree closure is >> moved to July 12th @ 12:00 EST. >> >> The community migration was completed successfully, however a problem >> related to how S3 and GCS handle gzip compressed files came up which will >> likely cause some issues in Firefox CI (read below for details). Since June >> 28th is preceding a long weekend followed by the Firefox 115 release, we've >> decided to delay the migration until July 12th, when we'll be under no time >> pressure to fix fallout. >> >> >> >> *Issue Details* >> It turns out that S3 serves the file with whatever encoding it was >> uploaded with no matter what. So if you request a gzip compressed artifact, >> S3 will serve you that gzip compressed artifact. GCS on the other hand, >> pays attention to the Accept-Encoding >> <https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding> >> header. So if you request a gzip compressed artifact, GCS will serve you >> the gzip compressed artifact if you specify `Accept-Encoding: gzip`, but >> otherwise helpfully decompress the artifact (as the lack of the >> `Accept-Encoding` header indicates you can't handle gzipped files). This >> means that after the transition, places that were expecting downloaded >> artifacts to be gzipped, may instead be decompressed. >> >> We'll use the extra time to perform extra testing and validation to try >> and root out issues ahead of time. However, it is not possible to find >> every place that downloads artifacts from Taskcluster. If you maintain any >> tasks or tools that download gzipped Taskcluster artifacts and want to >> avoid any disruption, please ensure you are passing the `Accept-Encoding: >> gzip` header in your GET requests. >> >> - Andrew >> >> >> >> On Tue, Jun 20, 2023 at 12:32 PM Andrew Halberstadt <[email protected]> >> wrote: >> >>> Hey everyone, >>> >>> We're planning a tree closure to migrate Firefox CI's artifact storage >>> backend from Amazon S3 to Google Cloud Storage: >>> >>> *When:* June 28th @ 12:00 EST >>> *Duration:* 2hrs (or longer if we need to rollback) >>> *What to expect:* >>> Try will remain open. While we anticipate the transition should be >>> mostly seamless, you may notice some recent artifacts will be missing for a >>> short period of time following the cutover (but they will be synced over >>> soon) and some unlucky tasks may fail with an exception and need to be >>> rerun. >>> >>> We'll also be migrating the Community Taskcluster instance tomorrow in >>> order to detect any issues that arise beforehand. If you notice missing >>> artifacts or other issues, please reach out on #firefox-ci >>> <https://matrix.to/#/%23firefox-ci:mozilla.org> (for issues with the >>> Firefox CI instance) or #taskcluster >>> <https://matrix.to/#/%23taskcluster:mozilla.org> (for issues with the >>> Community instance). >>> >>> Big thanks to Eric Maydeck for all the prerequisite work to make the >>> migration possible! >>> >>> Please reach out if you have any questions or concerns. >>> Cheers, >>> Andrew >>> >> -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAAJAz%2B7d1QCDMxFAsTecTtN94WbRPSNfs%2BtzbGmSzKR6y9gbxA%40mail.gmail.com.
