beratx opened a new issue, #6289: URL: https://github.com/apache/incubator-devlake/issues/6289
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened Similarly to #5385 the blueprint's Gitlab task fails due to missing Gitlab pipeline jobs. These jobs are belonging to Gitlab pipelines that are removed from Gitlab. So the task fails instead of ignoring it and continuing. <img width="1158" alt="Screenshot 2023-10-19 at 17 14 53" src="https://github.com/apache/incubator-devlake/assets/9766069/b1c57a98-d1ed-4ff3-a06e-3aa4e1868f9f";> Failed task's logs are showing that it attempts to collect some jobs but receives 404: ``` time="2023-10-19 12:42:35" level=warning msg=" [pipeline service] [pipeline #2202] [task #36327] [api async client] retry #0 calling projects/7865/jobs/9567773 caused by: Http DoAsync error calling [GET projects/7865/jobs/9567773]. Response: {"message":"404 Not found"} (404)" time="2023-10-19 12:42:37" level=info msg=" [pipeline service] [pipeline #2202] [task #36327] [collectApiJobs] finished records: 2278" time="2023-10-19 12:42:41" level=warning msg=" [pipeline service] [pipeline #2202] [task #36327] [api async client] retry #1 calling projects/7865/jobs/9567773 caused by: Http DoAsync error calling [GET projects/7865/jobs/9567773]. Response: {"message":"404 Not found"} (404)" time="2023-10-19 12:42:41" level=info msg=" [pipeline service] [pipeline #2202] [task #36327] [collectApiJobs] finished records: 2287" time="2023-10-19 12:42:44" level=info msg=" [pipeline service] [pipeline #2202] [task #36327] [collectApiJobs] finished records: 2296" time="2023-10-19 12:42:45" level=warning msg=" [pipeline service] [pipeline #2202] [task #36327] [api async client] retry #2 calling projects/7865/jobs/9567773 caused by: Http DoAsync error calling [GET projects/7865/jobs/9567773]. Response: {"message":"404 Not found"} (404)" time="2023-10-19 12:42:47" level=info msg=" [pipeline service] [pipeline #2202] [task #36327] [collectApiJobs] finished records: 2304" time="2023-10-19 12:42:51" level=error msg=" [pipeline service] [pipeline #2202] [task #36327] [api async client] Retry exceeded 3 times calling projects/7865/jobs/9567773. The last error was: Http DoAsync error calling [GET projects/7865/jobs/9567773]. Response: {"message":"404 Not found"} (404) Wraps: (2) Http DoAsync error calling [GET projects/7865/jobs/9567773]. Response: {"message":"404 Not found"} (404) Error types: (1) *hintdetail.withDetail (2) *errors.errorString" time="2023-10-19 12:42:51" level=info msg=" [pipeline service] [pipeline #2202] [task #36327] [collectApiJobs] finished records: 2313" time="2023-10-19 12:42:54" level=info msg=" [pipeline service] [pipeline #2202] [task #36327] [collectApiJobs] finished records: 2322" time="2023-10-19 12:42:55" level=error msg=" [pipeline service] [pipeline #2202] [task #36327] subtask collectApiJobs ended unexpectedly Wraps: (2) Retry exceeded 3 times calling projects/7865/jobs/9567773. The last error was: Http DoAsync error calling [GET projects/7865/jobs/9567773]. Response: {"message":"404 Not found"} (404) Error types: (1) *hintdetail.withDetail (2) *errors.errorString" ``` ### What do you expect to happen The pipeline should just ignore missing jobs/pipelines which are 404'd by gitlab and continue its information gathering - instead of completely failing. ### How to reproduce 1. Create blueprint with Gitlab integration and add a repository 2. Launch a pipeline on the configured repository 3. Delete executed pipeline from Gitlab 4. Run Devlake's sync pipeline ### Anything else It happens each time the sync pipeline is executed for the given blueprint ### Version v0.18.0 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
