zahorniak opened a new issue, #9305: URL: https://github.com/apache/devlake/issues/9305
## What and why to refactor The CircleCI extractors `extractJobs` and `extractWorkflows` use the old `api.NewApiExtractor`. This extractor reads **all** rows of the raw table on every run, also on incremental runs. The GitHub, GitLab, Jira and YouTrack plugins already use `api.NewStatefulApiExtractor`, which reads only the raw rows added since the last successful run. So the time for each run grows with the size of the raw table, not with the new CI activity. In our instance: | | | | --- | --- | | `_raw_circleci_api_jobs` | about 9.3M rows (10.3 GB) | | New raw rows per night | about 166k (almost all from the problem in #9304) | | `extractJobs` speed | about 490 rows/s with default MySQL settings | | `extractJobs` time per run | about 4.5 hours for one project, every night | The job extraction writes one row per commit. It uses almost no CPU on the DevLake or MySQL side; the disk sync on each commit sets the speed. Environment: DevLake **v1.0.3-beta14** (the code is the same on `main` at `351d86738`), MySQL 8.4, CircleCI Cloud, nightly incremental blueprint. ## Describe the solution you'd like Change `ExtractJobs` (`backend/plugins/circleci/tasks/job_extractor.go`) and `ExtractWorkflows` (`backend/plugins/circleci/tasks/workflow_extractor.go`) to `api.NewStatefulApiExtractor`, the same way as other plugins (for example `backend/plugins/github/tasks/pr_extractor.go`). - On an incremental run, the extractor reads only raw rows with `created_at >= since`. - On a full sync, or when the configuration changes, it reads all raw rows, as it does now. - Duplicate raw rows for the same job are safe: the tool-layer key is `(connection_id, workflow_id, id)` for jobs and `(connection_id, id)` for workflows, so a newer row replaces the older one. - Add or update the e2e tests for the two extractors. With #9304 fixed and this change, `extractJobs` in our instance would read about 20k rows per night, not 9.3M. I can start work on a PR for this if the DevLake maintainers approve this approach. ## Related issues - #9304: the "unfinished details" collectors collect the same jobs again without a limit. - #8907, #8956: earlier fixes in the same collectors. ## Additional context A one-time cleanup of the duplicate raw rows is not part of this change. After both fixes, a full sync removes the old raw rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
