dosubot[bot] commented on issue #8819: URL: https://github.com/apache/incubator-devlake/issues/8819#issuecomment-4161070743
<!-- Greeting --> Hi @squatboy! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the incubator-devlake team. <!-- Answer --> This is a valid gap. After digging into the code, here's what's happening and a couple of additional things worth noting for whoever picks up the fix: **Current behavior after PR #8637:** The `fetchAsync()` in [`graphql_collector.go`](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go#L276-L284) now catches `"Could not resolve to an Issue"` errors and logs a warning instead of crashing the task — but the stale row in `_tool_github_issues` is never cleaned up. The Phase 2 refresh path queries all local issues with `state='OPEN'` [[1]](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/issue_collector.go#L149-L161), so the orphaned issue keeps appearing in every future refresh cycle indefinitely. **The `_tool_github_issues` model has no tombstone or soft-delete fields** — no `deleted_at`, `is_stale`, etc. [[2]](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github/models/issue.go), so a fix would need to either hard-delete the row, update `State` to `CLOSED` (or a new sentinel value), or add a new field. **Batch-collateral issue:** There's a subtle additional problem — after the `dataErrors` loop in `fetchAsync()`, there's an early `return` [[3]](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go#L281). This means if *one* issue in a batch of 100 triggers `"Could not resolve"`, the `ResponseParser` is **skipped for the entire batch**, so valid issues in the same batch also miss their refresh update for that cycle. A fix would likely need to address at least two things: 1. **Stale row cleanup** — when the error is detected, delete or mark the corresponding `_tool_github_issues` row so it's excluded from future Phase 2 refresh queries. 2. **Batch handling** — avoid the early `return` so that valid issues in the same batch still get processed by `ResponseParser`. One approach could be to set `IgnoreQueryErrors: true` (like [`account_collector.go` and `job_collector.go`](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go#L82) already do) and handle the error inside the `ResponseParser` instead, or to filter out the errored entries and continue processing the rest of the batch. Since you indicated you're willing to submit a PR — the key files to look at are [`backend/plugins/github_graphql/tasks/issue_collector.go`](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/issue_collector.go) (Phase 2 refresh logic) and [`backend/helpers/pluginhelper/api/graphql_collector.go`](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go) (the `fetchAsync` error handling). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=brand-link).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=great_response&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=irrelevant_answer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=incorrect_sources&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=too_verbose&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e -4cbb-8856-8e54f9e7ef56?feedback_type=hallucination&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=bug_report&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-bug_report) | [Other](https://app.dosu.dev/response-feedback/bd7fc0a5-7a2e-4cbb-8856-8e54f9e7ef56?feedback_type=other&utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=feedback-other)</sup> [](https://app.dosu.dev/b4e8e847-d479-4541-83a8-d88d83fea5c9/ask?utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=chat-badge) [](https://cursor.com/link/prompt?text=This%20is%20a%20valid%20gap.%20After%20digging%20into%20the%20code%2C%20here%27s%20what%27s%20happening%20and%20a%20couple%20of%20additional%20things%20worth%20noting%20for%20whoever%20picks%20up%20the%20fix%3A%0A%0A%2A%2ACurrent%20behavior%20after%20PR%20%238637%3A%2A%2A%20The%20%60fetchAsync%28%29%60%20in%20%5B%60graphql_collector.go%60%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go%23L276-L284%29%20now%20catches%20%60%26quot%3BCould%20not%20resolve%20to%20an%20Issue%26quot%3B%60%20errors%20and%20logs%20a%20warning%20instead%20of%20crashing%20the%20task%20%E2%80%94%20bu t%20the%20stale%20row%20in%20%60_tool_github_issues%60%20is%20never%20cleaned%20up.%20The%20Phase%202%20refresh%20path%20queries%20all%20local%20issues%20with%20%60state%3D%27OPEN%27%60%20%5B%5B1%5D%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/issue_collector.go%23L149-L161%29%2C%20so%20the%20orphaned%20issue%20keeps%20appearing%20in%20every%20future%20refresh%20cycle%20indefinitely.%0A%0A%2A%2AThe%20%60_tool_github_issues%60%20model%20has%20no%20tombstone%20or%20soft-delete%20fields%2A%2A%20%E2%80%94%20no%20%60deleted_at%60%2C%20%60is_stale%60%2C%20etc.%20%5B%5B2%5D%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github/models/issue.go%29%2C%20so%20a%20fix%20would%20need%20to%20either%20hard-delete%20the%20row%2C%20update%20%60State%60%20to%20%60CLOSED%60%20%28or%20a%20new%20sentinel%20value%29%2C%20or%20add%20a%20new%20field.%0A% 0A%2A%2ABatch-collateral%20issue%3A%2A%2A%20There%27s%20a%20subtle%20additional%20problem%20%E2%80%94%20after%20the%20%60dataErrors%60%20loop%20in%20%60fetchAsync%28%29%60%2C%20there%27s%20an%20early%20%60return%60%20%5B%5B3%5D%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go%23L281%29.%20This%20means%20if%20%2Aone%2A%20issue%20in%20a%20batch%20of%20100%20triggers%20%60%26quot%3BCould%20not%20resolve%26quot%3B%60%2C%20the%20%60ResponseParser%60%20is%20%2A%2Askipped%20for%20the%20entire%20batch%2A%2A%2C%20so%20valid%20issues%20in%20the%20same%20batch%20also%20miss%20their%20refresh%20update%20for%20that%20cycle.%0A%0AA%20fix%20would%20likely%20need%20to%20address%20at%20least%20two%20things%3A%0A%0A1.%20%2A%2AStale%20row%20cleanup%2A%2A%20%E2%80%94%20when%20the%20error%20is%20detected%2C%20delete%20or%20mark%20the%20corresponding%20%60_tool_github_issues%60%20row%20so%20it%27s%20excl uded%20from%20future%20Phase%202%20refresh%20queries.%0A2.%20%2A%2ABatch%20handling%2A%2A%20%E2%80%94%20avoid%20the%20early%20%60return%60%20so%20that%20valid%20issues%20in%20the%20same%20batch%20still%20get%20processed%20by%20%60ResponseParser%60.%20One%20approach%20could%20be%20to%20set%20%60IgnoreQueryErrors%3A%20true%60%20%28like%20%5B%60account_collector.go%60%20and%20%60job_collector.go%60%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go%23L82%29%20already%20do%29%20and%20handle%20the%20error%20inside%20the%20%60ResponseParser%60%20instead%2C%20or%20to%20filter%20out%20the%20errored%20entries%20and%20continue%20processing%20the%20rest%20of%20the%20batch.%0A%0ASince%20you%20indicated%20you%27re%20willing%20to%20submit%20a%20PR%20%E2%80%94%20the%20key%20files%20to%20look%20at%20are%20%5B%60backend/plugins/github_graphql/tasks/issue_collector.go%60%5D%28https%3A//github.com/apach e/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/github_graphql/tasks/issue_collector.go%29%20%28Phase%202%20refresh%20logic%29%20and%20%5B%60backend/helpers/pluginhelper/api/graphql_collector.go%60%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/graphql_collector.go%29%20%28the%20%60fetchAsync%60%20error%20handling%29.) [](https://go.dosu.dev/discord-bot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-assistant&utm_content=join-discord) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/incubator-devlake/issues/8819) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
