ClaudioMascaro opened a new pull request, #8240:
URL: https://github.com/apache/incubator-devlake/pull/8240
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
### ⚠️ Pre Checklist
> Please complete _ALL_ items in this checklist, and remove before submitting
- [x] I have read through the [Contributing
Documentation](https://devlake.apache.org/community/).
- [ ] I have added relevant tests.
- [ ] I have added relevant documentation.
- [x] I will add labels to the PR, such as `pr-type/bug-fix`,
`pr-type/feature-development`, etc.
<!--
Thanks for submitting a pull request!
We appreciate you spending the time to work on these changes.
Please fill out as many sections below as possible.
-->
### Summary
Add pagination in github_graphql job collector task.
### Does this close any open issues?
Closes https://github.com/apache/incubator-devlake/issues/8028
### Screenshots
We needed to extract data from a large and complex repository, which has
over 30000 workflow runs, and some of that can have more than 200 job runs.
Whenever the Collect Job task started, It simply wouldn't finish the query
in time, entering in the retry flow:
we have tried to reduce api timeout, but it would only increase the number
of unsuccessful retries:
So, after implementing the solution, it would solve our case, and after 17
hours, it was able to collect all data:
(don't mind the log I added locally to debug)
as for comparison purposes, we have extracted data from a much less complex
repository, which before the implementation, took the following time:
and, after bringing off the solution (of course, rerunning in hard refresh
mode), there was no change in pipeline overall time:
### Other Information
Already merged on main branch, reopening as requested:
https://github.com/apache/incubator-devlake/pull/8233#pullrequestreview-2497839137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
