ansri-abodey opened a new issue, #8664: URL: https://github.com/apache/incubator-devlake/issues/8664
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement. ### Use case ### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement. ### Description I would like to contribute a new plugin to DevLake that ingests **GitHub Copilot Usage Metrics**. To fully understand the impact of AI on our development lifecycle, we need both high-level adoption trends and granular user-level data. I plan to use the **Copilot Usage Metrics API** (2022-11-28) to collect this data. ### Strategic Value: Why Measure This? Integrating Copilot metrics allows us to answer critical questions about the ROI of AI coding assistants by correlating them with existing DORA and engineering metrics: 1. **Velocity & Lead Time**: * *Hypothesis*: High Copilot usage (acceptance rates) should reduce **Lead Time for Changes** by speeding up the "coding" phase. * *Metric*: Correlate `total_lines_accepted` with `PR Cycle Time`. 2. **Quality & Stability**: * *Hypothesis*: We need to monitor if increased AI speed impacts code quality. * *Metric*: Correlate `copilot_active_users` with **Change Failure Rate** (CFR) and **Bug Count**. 3. **Adoption vs. Performance**: * *Hypothesis*: Teams with higher Copilot adoption perform better. * *Metric*: Compare **Deployment Frequency** between high-adoption and low-adoption teams (using the User-level breakdown). ### Data Sources I will target the following two report endpoints: 1. **Enterprise Metrics (Aggregated)** * Endpoint: `GET /enterprises/{enterprise}/copilot/metrics/reports/enterprise-1-day` * Purpose: Provides high-level daily summaries of active users, acceptance rates, and chat interactions across the entire enterprise. * Docs: [Get Copilot enterprise usage metrics](https://docs.github.com/en/enterprise-cloud@latest/rest/copilot/copilot-usage-metrics?apiVersion=2022-11-28#get-copilot-enterprise-usage-metrics) 2. **User Metrics (Granular)** * Endpoint: `GET /enterprises/{enterprise}/copilot/metrics/reports/users-1-day` * Purpose: Provides detailed usage stats per user (e.g., lines suggested vs. accepted, editor used, last activity). This allows for correlation with individual developer output. * Docs: [Get Copilot users usage metrics](https://docs.github.com/en/enterprise-cloud@latest/rest/copilot/copilot-usage-metrics?apiVersion=2022-11-28#get-copilot-users-usage-metrics) *Note: I will use the `*-1-day` endpoints to allow DevLake to incrementally collect and store daily historical data.* ### Implementation Plan I propose implementing this as a **Python Plugin** using the `pydevlake` framework. **Proposed Scope:** 1. Create a new Python plugin `copilot`. 2. **Stream 1: `CopilotEnterpriseMetricsStream`**: * Fetches the daily enterprise report URL. * Downloads and parses the JSON to store daily aggregated stats. 3. **Stream 2: `CopilotUserMetricsStream`**: * Fetches the daily user report URL. * Downloads and parses the JSON to store individual user activity. ### Questions for Maintainers 1. **Domain Modeling:** * Should the **Enterprise Metrics** map to a new table (e.g., `copilot_enterprise_daily`)? * Should the **User Metrics** map to a new table (e.g., `copilot_user_daily`)?### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement. ### Description I would like to contribute a new plugin to DevLake that ingests **GitHub Copilot Usage Metrics**. To fully understand the impact of AI on our development lifecycle, we need both high-level adoption trends and granular user-level data. I plan to use the **Copilot Usage Metrics API** (2022-11-28) to collect this data. ### Strategic Value: Why Measure This? Integrating Copilot metrics allows us to answer critical questions about the ROI of AI coding assistants by correlating them with existing DORA and engineering metrics: 1. **Velocity & Lead Time**: * *Hypothesis*: High Copilot usage (acceptance rates) should reduce **Lead Time for Changes** by speeding up the "coding" phase. * *Metric*: Correlate `total_lines_accepted` with `PR Cycle Time`. 2. **Quality & Stability**: * *Hypothesis*: We need to monitor if increased AI speed impacts code quality. * *Metric*: Correlate `copilot_active_users` with **Change Failure Rate** (CFR) and **Bug Count**. 3. **Adoption vs. Performance**: * *Hypothesis*: Teams with higher Copilot adoption perform better. * *Metric*: Compare **Deployment Frequency** between high-adoption and low-adoption teams (using the User-level breakdown). ### Data Sources I will target the following two report endpoints: 1. **Enterprise Metrics (Aggregated)** * Endpoint: `GET /enterprises/{enterprise}/copilot/metrics/reports/enterprise-1-day` * Purpose: Provides high-level daily summaries of active users, acceptance rates, and chat interactions across the entire enterprise. * Docs: [Get Copilot enterprise usage metrics](https://docs.github.com/en/enterprise-cloud@latest/rest/copilot/copilot-usage-metrics?apiVersion=2022-11-28#get-copilot-enterprise-usage-metrics) 2. **User Metrics (Granular)** * Endpoint: `GET /enterprises/{enterprise}/copilot/metrics/reports/users-1-day` * Purpose: Provides detailed usage stats per user (e.g., lines suggested vs. accepted, editor used, last activity). This allows for correlation with individual developer output. * Docs: [Get Copilot users usage metrics](https://docs.github.com/en/enterprise-cloud@latest/rest/copilot/copilot-usage-metrics?apiVersion=2022-11-28#get-copilot-users-usage-metrics) *Note: I will use the `*-1-day` endpoints to allow DevLake to incrementally collect and store daily historical data.* ### Implementation Plan I propose implementing this as a **Python Plugin** using the `pydevlake` framework. **Proposed Scope:** 1. Create a new Python plugin `copilot`. 2. **Stream 1: `CopilotEnterpriseMetricsStream`**: * Fetches the daily enterprise report URL. * Downloads and parses the JSON to store daily aggregated stats. 3. **Stream 2: `CopilotUserMetricsStream`**: * Fetches the daily user report URL. * Downloads and parses the JSON to store individual user activity. ### Questions for Maintainers 1. **Domain Modeling:** * Should the **Enterprise Metrics** map to a new table (e.g., `copilot_enterprise_daily`)? * Should the **User Metrics** map to a new table (e.g., `copilot_user_daily`)? ### Description _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
