Hi, I've looked into this issue and found that the default service account was removed during the weekend for some reason log viewer <https://pantheon.corp.google.com/logs/viewer?_ga=2.144495605.-1794192625.1569373214&project=apache-beam-testing&folder&organizationId=433637338589&minLogLevel=0&expandAll=false×tamp=2019-11-25T20:12:39.428000000Z&customFacets=&limitCustomFacetWidth=true&dateRangeStart=2019-11-18T20:41:06.480Z&dateRangeEnd=2019-11-25T20:41:06.480Z&interval=P7D&resource=gce_instance%2Finstance_id%2F8919707380078440409&scrollTimestamp=2019-11-25T07:55:19.038646702Z&advancedFilter=resource.type%3d%22service_account%22%0aresource.labels.email_id%3d%22844138762903-comp...@developer.gserviceaccount.com%22%0A%22DeleteServiceAccount%22&pinnedLogId=170pqn8e3ofe0&pinnedLogTimestamp=2019-11-25T07:55:19.038646702Z> . I restored the default service account. All workers should be backing to normal and jobs start passing now.
-yifan On Mon, Nov 25, 2019 at 11:17 AM Tomo Suzuki <suzt...@google.com> wrote: > Thank you for looking into this. > > On Mon, Nov 25, 2019 at 12:59 PM Yifan Zou <yifan...@google.com> wrote: > >> Greetings, >> >> We're seeing some tests encountering permission issues such as *'Failed >> to retrieve >> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-comp...@developer.gserviceaccount.com/token >> <http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-comp...@developer.gserviceaccount.com/token> >> from the Google Compute Enginemetadata service. Status: 404 >> Response:\nb\'"The service account was not found.'* >> >> I am looking onto it. We might need to reboot some build workers to >> restore the service account access. I'll try to make as little impact as >> possible on current running jobs. >> >> -yifan >> > > > -- > Regards, > Tomo >
