GitHub user dosubot[bot] edited a comment on the discussion: Question regarding scaling options.
<!-- Greeting --> Hi @DucTuLeAnh! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the incubator-devlake team. <!-- Answer --> You're correct — DevLake is designed as a [single-instance application](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/devops/deployment/k8s/k8s-deploy.yaml) and doesn't support horizontal scaling by increasing replicas. All blueprints are [processed sequentially by a single runner](https://github.com/apache/incubator-devlake/issues/8448#issuecomment-2908796317), which becomes the primary bottleneck at enterprise scale. For 1000+ GitLab repositories, there have been community reports of significant challenges: - [Full collection cycles taking ~2 weeks](https://github.com/apache/incubator-devlake/issues/8448#issuecomment-2908796317) for 80+ projects - [MySQL hitting 100% CPU at ~1,000 projects](https://github.com/apache/incubator-devlake/issues/8260) even with 16 cores / 64 GB RAM - [Raw tables growing to 21+ GB](https://github.com/apache/incubator-devlake/issues/8523) with 2000 repos due to user data duplication **What helps at scale:** 1. **Database indexes** — Community-tested indexes [reduced DORA calculations from hours to minutes](https://github.com/apache/incubator-devlake/issues/8260) 2. **GitLab-specific optimization** — Set `GITLAB_SERVER_COLLECT_ALL_USERS=false` to collect members per-project instead of globally ([reduced collection time by ~70%](https://github.com/apache/incubator-devlake/issues/8283)) 3. **Memory allocation** — [Minimum 6 GB container memory](https://github.com/apache/incubator-devlake/discussions/8576#discussioncomment-14401772) recommended 4. **Collection strategy** — Use incremental collection, narrow date ranges, and consider `SKIP_COMMIT_STAT=true` / `SKIP_COMMIT_FILES=true` The honest answer is that at 1000+ repositories, you'll likely face performance challenges even with optimizations. Parallel runners are the [most requested architectural enhancement](https://github.com/apache/incubator-devlake/issues/8448#issuecomment-2908796317) for this reason. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=reply-with-mention&utm_term=apache%2Fincubator-devlake).* --- Docs are dead. Just use [Dosu](https://dosu.dev?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-tagline&utm_term=apache%2Fincubator-devlake). [](https://app.dosu.dev/response-feedback?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-feedback&utm_term=apache%2Fincubator-devlake#message_id=88d0ad01-730f-4d10-a428-a3fe23565acf) [](https://github.dosu.com/apache/incubator-devlake?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-ask-repo&utm_term=apache%2Fincubator-devlake) [](https://app.dosu.dev/signup?referrer=openSource&source=github-footer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-share-team&utm_term=apache%2Fincubator-devlake) GitHub link: https://github.com/apache/incubator-devlake/discussions/8802#discussioncomment-16276817 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
