GitHub user dosubot[bot] edited a comment on the discussion: Collect Pull requests Phase 2 triggers persistent 502s
<!-- Answer --> Great write-up — your analysis is spot-on. Here's what I found in the codebase and existing issues that directly relates: **The `updateRateRemaining` panic is a known bug tracked in [#8788](https://github.com/apache/incubator-devlake/issues/8788)** [[1]](https://github.com/apache/incubator-devlake/issues/8788). The panic at `graphql_async_client.go:129` crashes the process on *any* error from the rate-limit polling goroutine — including transient 502s and 401s from expired GitHub App tokens. There's actually a second panic at [line 71](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_async_client.go#L71) during initial client setup as well. **[PR #8791](https://github.com/apache/incubator-devlake/pull/8791)** [[2]](https://github.com/apache/incubator-devlake/pull/8791) provides the comprehensive fix for this — it replaces both panics with graceful error handling and also wires up the GraphQL client to use `RefreshRoundTripper` for automatic GitHub App token renewal (matching what the REST client already does via #8746). However, **PR #8791 has been open since March 22 and is not yet merged**. You may want to comment on it to signal urgency given your 42+ pod restarts. **On the `InputStep: 100` issue** — you're correct that it's [hardcoded at line 243](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/pr_collector.go#L243) in `pr_collector.go` with no configuration option. The same value of 100 is used for [issue collection](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/issue_collector.go#L166) and [account collection](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/account_collector.go#L102). Notably, the **job collector already uses `InputStep: 10`** [[3]](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/job_collector.go#L191), likely for the exact same reason — large responses causing 502s. A similar problem was addressed for jobs i n [PR #8616](https://github.com/apache/incubator-devlake/pull/8616) which introduced the `GITHUB_JOB_GRAPHQL_PAGE_SIZE` env var, but no equivalent was added for PRs or issues. Your suggestion #1 (reducing `InputStep` to 10-20) is the right approach. Ideally this would follow the same pattern as `GITHUB_JOB_GRAPHQL_PAGE_SIZE` — an environment variable like `GITHUB_PR_GRAPHQL_BATCH_SIZE` — so it can be tuned per deployment without forking the code. **One additional workaround until these fixes land:** @zaiddialpad reported very [similar 502 issues at scale](https://github.com/apache/incubator-devlake/discussions/8821#discussioncomment-16398435) in discussion #8821. The recommended mitigations are: - Set `API_RETRY=5` or higher (default is 3) - Increase `API_TIMEOUT` for more breathing room between retries - If using GitHub App auth, try switching to a **Personal Access Token** to avoid the token expiry → panic chain entirely - Split large blueprints so individual runs stay under ~45 minutes **On making Phase 2 optional:** There's no direct toggle, but setting `fullSync: true` in the Blueprint's SyncPolicy skips Phase 2 since it only runs during incremental collection [[4]](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/api_collector_stateful.go#L73-L84). The trade-off is you lose incremental behavior entirely, which is probably not what you want for a daily sync. A dedicated per-scope flag to skip Phase 2 refetch would be the ideal long-term solution. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=reply-with-mention&utm_term=apache%2Fincubator-devlake).* --- Docs are dead. Just use [Dosu](https://dosu.dev?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-tagline&utm_term=apache%2Fincubator-devlake). [](https://app.dosu.dev/response-feedback?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-feedback&utm_term=apache%2Fincubator-devlake#message_id=238f958d-5a2e-4721-b8a2-e449588c2d97) [](https://github.dosu.com/apache/incubator-devlake?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-ask-repo&utm_term=apache%2Fincubator-devlake) [](https://app.dosu.dev/signup?referrer=openSource&source=github-footer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-share-team&utm_term=apache%2Fincubator-devlake) GitHub link: https://github.com/apache/incubator-devlake/discussions/8824#discussioncomment-16478979 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
