This is an automated email from the ASF dual-hosted git repository.
klesh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-devlake.git
The following commit(s) were added to refs/heads/main by this push:
new 611556cd0 feat: GitHub App token refresh (#8746)
611556cd0 is described below
commit 611556cd0f3e4dd8a1f2df257e2d7602dabb301a
Author: Leif Roger Frøysaa <[email protected]>
AuthorDate: Thu Mar 12 13:48:10 2026 +0100
feat: GitHub App token refresh (#8746)
* feat(github): auto-refresh GitHub App installation tokens
Add transport-level token refresh for GitHub App (AppKey) connections.
GitHub App installation tokens expire after ~1 hour; this adds proactive
refresh (before expiry) and reactive refresh (on 401) using the existing
TokenProvider/RefreshRoundTripper infrastructure.
New files:
- app_installation_refresh.go: refresh logic + DB persistence
- refresh_api_client.go: minimal ApiClient for token refresh POST
- cmd/test_refresh/main.go: manual test script for real GitHub Apps
Modified:
- connection.go: export GetInstallationAccessToken, parse ExpiresAt
- token_provider.go: add refreshFn for pluggable refresh strategies
- round_tripper.go: document dual Authorization header interaction
- api_client.go: wire AppKey connections into refresh infrastructure
- Tests updated for new constructors and AppKey refresh flow
* feat(github): add diagnostic logging to GitHub App token refresh
Add structured logging at key decision points for token refresh:
- Token provider creation (connection ID, installation ID, expiry)
- Round tripper installation (connection ID, auth method)
- Proactive refresh trigger (near-expiry detection)
- Refresh start/success/failure (old/new token prefixes, expiry times)
- DB persistence success/failure
- Reactive 401 refresh and skip-due-to-concurrent-refresh
All logs route through the DevLake logger to pipeline log files.
* fix(github): prevent deadlock and fix token persistence in App token
refresh
Deadlock fix: NewAppInstallationTokenProvider now captures client.Transport
(the base transport) before wrapping with RefreshRoundTripper. The refresh
function uses newRefreshApiClientWithTransport(baseTransport) to POST for
new installation tokens, bypassing the RefreshRoundTripper entirely.
Token persistence fix: PersistEncryptedTokenColumns() manually encrypts
tokens via plugin.Encrypt() then writes ciphertext via dal.UpdateColumns
with conn.TableName() (a string) as the first argument. Passing the table
name string makes GORM use Table() instead of Model(), preventing the
encdec serializer from corrupting the in-memory token value. The encryption
secret is threaded from taskCtx.GetConfig(ENCRYPTION_SECRET) through
CreateApiClient to TokenProvider to persist functions.
Also persists the initial App token at startup for DB consistency, and
adds TestProactiveRefreshNoDeadlock with a real RSA key to verify the
deadlock scenario is resolved.
* fix(grafana): update dashboard descriptions to list all supported data
sources (#8741)
Several dashboard introduction panels hardcoded "GitHub and Jira" as
required data sources, even though the underlying queries use generic
domain layer tables that work with any supported Git tool or issue
tracker. Updated to list all supported sources following the pattern
already used by DORA and WorkLogs dashboards.
Closes #8740
Co-authored-by: Spiff Azeta <[email protected]>
* fix: modify cicd_deployments name from varchar to text (#8724)
* fix: modify cicd_deployments name from varchar to text
* fix: update the year
* fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL
methods for MySQL 8.x compatibility (#8745)
* fix(azuredevops): default empty entities and add CROSS to repo scope in
makeScopeV200 (#8751)
When scopeConfig.Entities is empty (common when no entities are
explicitly selected in the UI), makeScopeV200 produced zero scopes,
leaving project_mapping with no rows. Additionally, the repo scope
condition did not check for DOMAIN_TYPE_CROSS, so selecting only
CROSS would not create a repo scope, breaking DORA metrics.
This adds the same fixes applied to GitLab in #8743.
Closes #8749
* fix(bitbucket): default empty entities to all domain types in
makeScopesV200 (#8750)
When scopeConfig.Entities is empty (common when no entities are
explicitly selected in the UI), makeScopesV200 produced zero scopes,
leaving project_mapping with no repo rows. This adds the same
empty-entities default applied to GitLab in #8743.
Closes #8748
* fix(github): remove unused refresh client constructor and update tests
---------
Co-authored-by: Spiff Azeta <[email protected]>
Co-authored-by: Spiff Azeta <[email protected]>
Co-authored-by: Dan Crews <[email protected]>
Co-authored-by: Tomoya Kawaguchi
<[email protected]>
---
backend/plugins/github/models/connection.go | 21 +-
backend/plugins/github/tasks/api_client.go | 32 ++-
.../github/token/app_installation_refresh.go | 152 +++++++++++++
.../plugins/github/token/cmd/test_refresh/main.go | 240 +++++++++++++++++++++
backend/plugins/github/token/refresh_api_client.go | 115 ++++++++++
.../github/token/refresh_api_client_test.go | 106 +++++++++
backend/plugins/github/token/round_tripper.go | 5 +
backend/plugins/github/token/round_tripper_test.go | 193 ++++++++++++++++-
backend/plugins/github/token/token_provider.go | 102 +++++++--
.../plugins/github/token/token_provider_test.go | 117 +++++++---
10 files changed, 1019 insertions(+), 64 deletions(-)
diff --git a/backend/plugins/github/models/connection.go
b/backend/plugins/github/models/connection.go
index d03353c18..51034f67c 100644
--- a/backend/plugins/github/models/connection.go
+++ b/backend/plugins/github/models/connection.go
@@ -81,13 +81,15 @@ func (conn *GithubConn) PrepareApiClient(apiClient
plugin.ApiClient) errors.Erro
}
if conn.AuthMethod == AppKey && conn.InstallationID != 0 {
- token, err := conn.getInstallationAccessToken(apiClient)
+ token, err := conn.GetInstallationAccessToken(apiClient)
if err != nil {
return err
}
-
- conn.Token = token.Token
- conn.tokens = []string{token.Token}
+ var expiresAt *time.Time
+ if !token.ExpiresAt.IsZero() {
+ expiresAt = &token.ExpiresAt
+ }
+ conn.UpdateToken(token.Token, "", expiresAt, nil)
}
return nil
@@ -354,7 +356,8 @@ type GithubUserOfToken struct {
}
type InstallationToken struct {
- Token string `json:"token"`
+ Token string `json:"token"`
+ ExpiresAt time.Time `json:"expires_at"`
}
type GithubApp struct {
@@ -397,7 +400,7 @@ func (gak *GithubAppKey) CreateJwt() (string, errors.Error)
{
return tokenString, nil
}
-func (gak *GithubAppKey) getInstallationAccessToken(
+func (gak *GithubAppKey) GetInstallationAccessToken(
apiClient plugin.ApiClient,
) (*InstallationToken, errors.Error) {
@@ -417,12 +420,18 @@ func (gak *GithubAppKey) getInstallationAccessToken(
if err != nil {
return nil, err
}
+ if resp.StatusCode != http.StatusCreated && resp.StatusCode !=
http.StatusOK {
+ return nil,
errors.HttpStatus(resp.StatusCode).New(fmt.Sprintf("unexpected status code
while getting installation access token: %s", string(body)))
+ }
var installationToken InstallationToken
err = errors.Convert(json.Unmarshal(body, &installationToken))
if err != nil {
return nil, err
}
+ if installationToken.Token == "" {
+ return nil, errors.Default.New("empty installation access token
returned")
+ }
return &installationToken, nil
}
diff --git a/backend/plugins/github/tasks/api_client.go
b/backend/plugins/github/tasks/api_client.go
index 268af8ece..42181ff13 100644
--- a/backend/plugins/github/tasks/api_client.go
+++ b/backend/plugins/github/tasks/api_client.go
@@ -35,14 +35,18 @@ func CreateApiClient(taskCtx plugin.TaskContext, connection
*models.GithubConnec
return nil, err
}
- // Inject TokenProvider if refresh token is present
- if connection.RefreshToken != "" {
- logger := taskCtx.GetLogger()
- db := taskCtx.GetDal()
-
- // Create TokenProvider
- tp := token.NewTokenProvider(connection, db,
apiClient.GetClient(), logger)
+ logger := taskCtx.GetLogger()
+ db := taskCtx.GetDal()
+ encryptionSecret := taskCtx.GetConfig(plugin.EncodeKeyEnvStr)
+ // Inject TokenProvider for OAuth refresh or GitHub App installation
tokens.
+ var tp *token.TokenProvider
+ if connection.RefreshToken != "" {
+ tp = token.NewTokenProvider(connection, db,
apiClient.GetClient(), logger, encryptionSecret)
+ } else if connection.AuthMethod == models.AppKey &&
connection.InstallationID != 0 {
+ tp = token.NewAppInstallationTokenProvider(connection, db,
apiClient.GetClient(), logger, encryptionSecret)
+ }
+ if tp != nil {
// Wrap the transport
baseTransport := apiClient.GetClient().Transport
if baseTransport == nil {
@@ -51,6 +55,20 @@ func CreateApiClient(taskCtx plugin.TaskContext, connection
*models.GithubConnec
rt := token.NewRefreshRoundTripper(baseTransport, tp)
apiClient.GetClient().Transport = rt
+ logger.Info("Installed token refresh round tripper for
connection %d (authMethod=%s)",
+ connection.ID, connection.AuthMethod)
+ }
+
+ // Persist the freshly minted token so the DB has a correctly encrypted
value.
+ // PrepareApiClient (called by NewApiClientFromConnection) mints the
token
+ // in-memory but does not persist it; without this, the DB may contain
a stale
+ // or corrupted token that breaks GET /connections.
+ if connection.AuthMethod == models.AppKey && connection.Token != "" {
+ if err := token.PersistEncryptedTokenColumns(db, connection,
encryptionSecret, logger, false); err != nil {
+ logger.Warn(err, "Failed to persist initial token for
connection %d", connection.ID)
+ } else {
+ logger.Info("Persisted initial token for connection
%d", connection.ID)
+ }
}
// create rate limit calculator
diff --git a/backend/plugins/github/token/app_installation_refresh.go
b/backend/plugins/github/token/app_installation_refresh.go
new file mode 100644
index 000000000..2f4cf04b8
--- /dev/null
+++ b/backend/plugins/github/token/app_installation_refresh.go
@@ -0,0 +1,152 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package token
+
+import (
+ "time"
+
+ "github.com/apache/incubator-devlake/core/dal"
+ "github.com/apache/incubator-devlake/core/errors"
+ "github.com/apache/incubator-devlake/core/log"
+ "github.com/apache/incubator-devlake/core/plugin"
+ "github.com/apache/incubator-devlake/plugins/github/models"
+)
+
+// tokenPrefix returns the first n characters of a token for safe logging.
+func tokenPrefix(token string, n int) string {
+ if len(token) <= n {
+ return token
+ }
+ return token[:n] + "..."
+}
+
+func refreshGitHubAppInstallationToken(tp *TokenProvider) errors.Error {
+ if tp == nil || tp.conn == nil {
+ return errors.Default.New("missing github connection for app
token refresh")
+ }
+ if tp.conn.AuthMethod != models.AppKey || tp.conn.InstallationID == 0 {
+ return errors.Default.New("invalid github app connection for
token refresh")
+ }
+ if tp.conn.Endpoint == "" {
+ return errors.Default.New("missing github endpoint for token
refresh")
+ }
+
+ oldToken := tp.conn.Token
+ if tp.logger != nil {
+ expiresStr := "unknown"
+ if tp.conn.TokenExpiresAt != nil {
+ expiresStr = tp.conn.TokenExpiresAt.Format(time.RFC3339)
+ }
+ tp.logger.Info(
+ "Refreshing GitHub App installation token for
connection %d (installation %d), old token=%s, expires_at=%s",
+ tp.conn.ID, tp.conn.InstallationID,
+ tokenPrefix(oldToken, 8),
+ expiresStr,
+ )
+ }
+
+ // Use baseTransport (the unwrapped transport) to avoid deadlock.
+ // The httpClient's transport may be the RefreshRoundTripper which would
+ // re-enter GetToken() and deadlock on the mutex.
+ apiClient := newRefreshApiClientWithTransport(tp.conn.Endpoint,
tp.baseTransport)
+ installationToken, err :=
tp.conn.GithubAppKey.GetInstallationAccessToken(apiClient)
+ if err != nil {
+ if tp.logger != nil {
+ tp.logger.Error(err, "Failed to refresh GitHub App
installation token for connection %d", tp.conn.ID)
+ }
+ return err
+ }
+
+ var expiresAt *time.Time
+ if !installationToken.ExpiresAt.IsZero() {
+ expiresAt = &installationToken.ExpiresAt
+ }
+ tp.conn.UpdateToken(installationToken.Token, "", expiresAt, nil)
+
+ if tp.logger != nil {
+ tp.logger.Info(
+ "Successfully refreshed GitHub App installation token
for connection %d, new token=%s, new expires_at=%s",
+ tp.conn.ID,
+ tokenPrefix(installationToken.Token, 8),
+ installationToken.ExpiresAt.Format(time.RFC3339),
+ )
+ }
+
+ persistAppToken(tp.dal, tp.conn, tp.encryptionSecret, tp.logger)
+ return nil
+}
+
+func persistAppToken(d dal.Dal, conn *models.GithubConnection,
encryptionSecret string, logger log.Logger) {
+ if d == nil || conn == nil {
+ return
+ }
+ if err := PersistEncryptedTokenColumns(d, conn, encryptionSecret,
logger, false); err != nil {
+ if logger != nil {
+ logger.Warn(err, "Failed to persist refreshed app
installation token for connection %d", conn.ID)
+ }
+ } else if logger != nil {
+ logger.Info("Persisted refreshed app installation token for
connection %d", conn.ID)
+ }
+}
+
+// PersistEncryptedTokenColumns manually encrypts token fields and writes them
+// to the DB using UpdateColumns (map-based), which only touches the specified
+// columns. This avoids two problems:
+// - dal.Update (GORM Save) writes ALL fields, including
refresh_token_expires_at
+// which may have Go zero time that MySQL rejects as '0000-00-00'.
+// - dal.UpdateColumns with plaintext bypasses the GORM encdec serializer,
+// writing unencrypted tokens that corrupt subsequent reads.
+//
+// IMPORTANT: We pass the table name string (not the conn struct) to
UpdateColumns
+// so that GORM uses Table() instead of Model(). When Model(conn) is used, GORM
+// processes the encdec serializer on the struct's Token field during statement
+// preparation, which overwrites conn.Token in memory with the encrypted
ciphertext.
+// This corrupts the in-memory token causing immediate 401s on the next API
call.
+//
+// If includeRefreshToken is true, refresh_token and refresh_token_expires_at
+// are also written (used by the OAuth refresh path where these values are
valid).
+func PersistEncryptedTokenColumns(d dal.Dal, conn *models.GithubConnection,
encryptionSecret string, logger log.Logger, includeRefreshToken bool)
errors.Error {
+ encToken, err := plugin.Encrypt(encryptionSecret, conn.Token)
+ if err != nil {
+ return errors.Default.Wrap(err, "failed to encrypt token for
persistence")
+ }
+
+ sets := []dal.DalSet{
+ {ColumnName: "token", Value: encToken},
+ {ColumnName: "token_expires_at", Value: conn.TokenExpiresAt},
+ }
+
+ if includeRefreshToken {
+ encRefreshToken, err := plugin.Encrypt(encryptionSecret,
conn.RefreshToken)
+ if err != nil {
+ return errors.Default.Wrap(err, "failed to encrypt
refresh_token for persistence")
+ }
+ sets = append(sets,
+ dal.DalSet{ColumnName: "refresh_token", Value:
encRefreshToken},
+ dal.DalSet{ColumnName: "refresh_token_expires_at",
Value: conn.RefreshTokenExpiresAt},
+ )
+ }
+
+ // Use the table name string instead of the conn struct to prevent GORM
from
+ // running the encdec serializer on conn.Token during Model()
processing.
+ return d.UpdateColumns(
+ conn.TableName(),
+ sets,
+ dal.Where("id = ?", conn.ID),
+ )
+}
diff --git a/backend/plugins/github/token/cmd/test_refresh/main.go
b/backend/plugins/github/token/cmd/test_refresh/main.go
new file mode 100644
index 000000000..be1854dcc
--- /dev/null
+++ b/backend/plugins/github/token/cmd/test_refresh/main.go
@@ -0,0 +1,240 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// This script lists GitHub App installations and tests the token refresh flow.
+//
+// Usage:
+//
+// GITHUB_APP_ID=123456 GITHUB_APP_PEM="$(cat private-key.pem)" go run
./plugins/github/token/cmd/test_refresh/
+//
+// Or if the key is in a file:
+//
+// GITHUB_APP_ID=123456 GITHUB_APP_PEM_FILE=/path/to/private-key.pem go
run ./plugins/github/token/cmd/test_refresh/
+package main
+
+import (
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "os"
+ "time"
+
+ "github.com/golang-jwt/jwt/v5"
+)
+
+type installation struct {
+ ID int `json:"id"`
+ Account struct {
+ Login string `json:"login"`
+ } `json:"account"`
+ AppID int `json:"app_id"`
+}
+
+type installationToken struct {
+ Token string `json:"token"`
+ ExpiresAt time.Time `json:"expires_at"`
+}
+
+func main() {
+ appID := os.Getenv("GITHUB_APP_ID")
+ if appID == "" {
+ fatal("GITHUB_APP_ID env var is required")
+ }
+
+ pemData := os.Getenv("GITHUB_APP_PEM")
+ if pemData == "" {
+ pemFile := os.Getenv("GITHUB_APP_PEM_FILE")
+ if pemFile == "" {
+ fatal("Set GITHUB_APP_PEM (contents) or
GITHUB_APP_PEM_FILE (path)")
+ }
+ data, err := os.ReadFile(pemFile)
+ if err != nil {
+ fatal("failed to read PEM file: %v", err)
+ }
+ pemData = string(data)
+ }
+
+ // Step 1: Create a JWT signed with the app's private key
+ fmt.Println("=== Step 1: Creating JWT from App ID and private key ===")
+ jwtToken, err := createJWT(appID, pemData)
+ if err != nil {
+ fatal("failed to create JWT: %v", err)
+ }
+ fmt.Printf("JWT created (first 20 chars): %s...\n\n", jwtToken[:20])
+
+ // Step 2: List installations
+ fmt.Println("=== Step 2: Listing installations for this app ===")
+ installations, err := listInstallations(jwtToken)
+ if err != nil {
+ fatal("failed to list installations: %v", err)
+ }
+ if len(installations) == 0 {
+ fatal("no installations found for this app")
+ }
+ for _, inst := range installations {
+ fmt.Printf(" Installation ID: %d Account: %s\n", inst.ID,
inst.Account.Login)
+ }
+ fmt.Println()
+
+ // Step 3: Get an installation token for the first installation
+ inst := installations[0]
+ fmt.Printf("=== Step 3: Minting installation token for %s (ID: %d)
===\n", inst.Account.Login, inst.ID)
+ token1, err := getInstallationToken(jwtToken, inst.ID)
+ if err != nil {
+ fatal("failed to get installation token: %v", err)
+ }
+ fmt.Printf("Token 1: %s... Expires: %s\n\n", token1.Token[:10],
token1.ExpiresAt.Format(time.RFC3339))
+
+ // Step 4: Make an API call with the token to verify it works
+ fmt.Println("=== Step 4: Verifying token works (GET
/installation/repositories) ===")
+ err = verifyToken(token1.Token)
+ if err != nil {
+ fatal("token verification failed: %v", err)
+ }
+ fmt.Printf("Token is valid and working.\n\n")
+
+ // Step 5: Simulate the refresh flow — mint a second token (as our
refreshFn would)
+ fmt.Println("=== Step 5: Simulating token refresh (minting a second
token) ===")
+ jwtToken2, err := createJWT(appID, pemData)
+ if err != nil {
+ fatal("failed to create second JWT: %v", err)
+ }
+ token2, err := getInstallationToken(jwtToken2, inst.ID)
+ if err != nil {
+ fatal("failed to get second installation token: %v", err)
+ }
+ fmt.Printf("Token 2: %s... Expires: %s\n", token2.Token[:10],
token2.ExpiresAt.Format(time.RFC3339))
+
+ if token1.Token == token2.Token {
+ fmt.Println("Note: Both tokens are identical (GitHub may cache
short-lived tokens)")
+ } else {
+ fmt.Println("Tokens are different — refresh produced a new
token.")
+ }
+ fmt.Println()
+
+ // Step 6: Verify the new token works
+ fmt.Println("=== Step 6: Verifying refreshed token works ===")
+ err = verifyToken(token2.Token)
+ if err != nil {
+ fatal("refreshed token verification failed: %v", err)
+ }
+ fmt.Println("Refreshed token is valid and working.")
+
+ fmt.Println("\n=== All steps passed. The token refresh flow works
correctly. ===")
+ fmt.Printf("\nFor reference, your Installation ID is: %d\n", inst.ID)
+}
+
+func createJWT(appID, pemData string) (string, error) {
+ privateKey, err := jwt.ParseRSAPrivateKeyFromPEM([]byte(pemData))
+ if err != nil {
+ return "", fmt.Errorf("invalid PEM key: %w", err)
+ }
+
+ now := time.Now().Unix()
+ token := jwt.NewWithClaims(jwt.SigningMethodRS256, jwt.MapClaims{
+ "iat": now,
+ "exp": now + (10 * 60), // 10 minutes
+ "iss": appID,
+ })
+
+ signed, err := token.SignedString(privateKey)
+ if err != nil {
+ return "", fmt.Errorf("failed to sign JWT: %w", err)
+ }
+ return signed, nil
+}
+
+func listInstallations(jwtToken string) ([]installation, error) {
+ req, err := http.NewRequest("GET",
"https://api.github.com/app/installations";, nil)
+ if err != nil {
+ return nil, err
+ }
+ req.Header.Set("Authorization", "Bearer "+jwtToken)
+ req.Header.Set("Accept", "application/vnd.github+json")
+
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ return nil, err
+ }
+ defer resp.Body.Close()
+
+ body, _ := io.ReadAll(resp.Body)
+ if resp.StatusCode != 200 {
+ return nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode,
string(body))
+ }
+
+ var result []installation
+ if err := json.Unmarshal(body, &result); err != nil {
+ return nil, fmt.Errorf("failed to parse response: %w", err)
+ }
+ return result, nil
+}
+
+func getInstallationToken(jwtToken string, installationID int)
(*installationToken, error) {
+ url :=
fmt.Sprintf("https://api.github.com/app/installations/%d/access_tokens";,
installationID)
+ req, err := http.NewRequest("POST", url, nil)
+ if err != nil {
+ return nil, err
+ }
+ req.Header.Set("Authorization", "Bearer "+jwtToken)
+ req.Header.Set("Accept", "application/vnd.github+json")
+
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ return nil, err
+ }
+ defer resp.Body.Close()
+
+ body, _ := io.ReadAll(resp.Body)
+ if resp.StatusCode != http.StatusCreated {
+ return nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode,
string(body))
+ }
+
+ var token installationToken
+ if err := json.Unmarshal(body, &token); err != nil {
+ return nil, fmt.Errorf("failed to parse token response: %w",
err)
+ }
+ return &token, nil
+}
+
+func verifyToken(token string) error {
+ req, err := http.NewRequest("GET",
"https://api.github.com/installation/repositories?per_page=1";, nil)
+ if err != nil {
+ return err
+ }
+ req.Header.Set("Authorization", "Bearer "+token)
+ req.Header.Set("Accept", "application/vnd.github+json")
+
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ return err
+ }
+ defer resp.Body.Close()
+
+ if resp.StatusCode != 200 {
+ body, _ := io.ReadAll(resp.Body)
+ return fmt.Errorf("HTTP %d: %s", resp.StatusCode, string(body))
+ }
+ fmt.Printf(" HTTP 200 OK (X-RateLimit-Remaining: %s)\n",
resp.Header.Get("X-RateLimit-Remaining"))
+ return nil
+}
+
+func fatal(format string, args ...interface{}) {
+ fmt.Fprintf(os.Stderr, "FATAL: "+format+"\n", args...)
+ os.Exit(1)
+}
diff --git a/backend/plugins/github/token/refresh_api_client.go
b/backend/plugins/github/token/refresh_api_client.go
new file mode 100644
index 000000000..1db6f6917
--- /dev/null
+++ b/backend/plugins/github/token/refresh_api_client.go
@@ -0,0 +1,115 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package token
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "net/http"
+ "net/url"
+ "time"
+
+ "github.com/apache/incubator-devlake/core/errors"
+ "github.com/apache/incubator-devlake/core/plugin"
+ "github.com/apache/incubator-devlake/helpers/pluginhelper/api"
+)
+
+type refreshApiClient struct {
+ endpoint string
+ client *http.Client
+ timeout time.Duration
+}
+
+// newRefreshApiClientWithTransport creates a refreshApiClient using a specific
+// http.RoundTripper instead of an existing *http.Client. This is critical for
+// avoiding deadlocks: the caller's *http.Client may have a RefreshRoundTripper
+// transport that re-enters GetToken() and deadlocks on the mutex.
+func newRefreshApiClientWithTransport(endpoint string, transport
http.RoundTripper) plugin.ApiClient {
+ if transport == nil {
+ transport = http.DefaultTransport
+ }
+ return &refreshApiClient{
+ endpoint: endpoint,
+ client: &http.Client{Transport: transport},
+ timeout: 10 * time.Second,
+ }
+}
+
+func (c *refreshApiClient) SetData(name string, data interface{}) {}
+
+func (c *refreshApiClient) GetData(name string) interface{} { return nil }
+
+func (c *refreshApiClient) SetHeaders(headers map[string]string) {}
+
+func (c *refreshApiClient) SetBeforeFunction(callback
plugin.ApiClientBeforeRequest) {}
+
+func (c *refreshApiClient) GetBeforeFunction() plugin.ApiClientBeforeRequest {
return nil }
+
+func (c *refreshApiClient) SetAfterFunction(callback
plugin.ApiClientAfterResponse) {}
+
+func (c *refreshApiClient) GetAfterFunction() plugin.ApiClientAfterResponse {
return nil }
+
+func (c *refreshApiClient) Get(path string, query url.Values, headers
http.Header) (*http.Response, errors.Error) {
+ return c.do(http.MethodGet, path, query, nil, headers)
+}
+
+func (c *refreshApiClient) Post(path string, query url.Values, body
interface{}, headers http.Header) (*http.Response, errors.Error) {
+ return c.do(http.MethodPost, path, query, body, headers)
+}
+
+func (c *refreshApiClient) do(method, path string, query url.Values, body
interface{}, headers http.Header) (*http.Response, errors.Error) {
+ uri, err := api.GetURIStringPointer(c.endpoint, path, query)
+ if err != nil {
+ return nil, err
+ }
+
+ var reqBody *bytes.Reader
+ if body != nil {
+ payload, err := json.Marshal(body)
+ if err != nil {
+ return nil, errors.Convert(err)
+ }
+ reqBody = bytes.NewReader(payload)
+ } else {
+ reqBody = bytes.NewReader(nil)
+ }
+
+ ctx, cancel := context.WithTimeout(context.Background(), c.timeout)
+ defer cancel()
+
+ req, err := errors.Convert01(http.NewRequestWithContext(ctx, method,
*uri, reqBody))
+ if err != nil {
+ return nil, err
+ }
+ if body != nil {
+ req.Header.Set("Content-Type", "application/json")
+ }
+
+ for name, values := range headers {
+ for _, value := range values {
+ req.Header.Add(name, value)
+ }
+ }
+
+ res, err := errors.Convert01(c.client.Do(req))
+ if err != nil {
+ return nil, err
+ }
+ return res, nil
+}
diff --git a/backend/plugins/github/token/refresh_api_client_test.go
b/backend/plugins/github/token/refresh_api_client_test.go
new file mode 100644
index 000000000..60cdbba2f
--- /dev/null
+++ b/backend/plugins/github/token/refresh_api_client_test.go
@@ -0,0 +1,106 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package token
+
+import (
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/assert"
+)
+
+func TestRefreshApiClientPost(t *testing.T) {
+ var receivedMethod string
+ var receivedPath string
+ var receivedAuth string
+
+ server := httptest.NewServer(http.HandlerFunc(func(w
http.ResponseWriter, r *http.Request) {
+ receivedMethod = r.Method
+ receivedPath = r.URL.Path
+ receivedAuth = r.Header.Get("Authorization")
+ w.WriteHeader(http.StatusCreated)
+
w.Write([]byte(`{"token":"ghs_test123","expires_at":"2026-03-02T12:00:00Z"}`))
+ }))
+ defer server.Close()
+
+ client := newRefreshApiClientWithTransport(server.URL,
server.Client().Transport)
+
+ headers := http.Header{
+ "Authorization": []string{"Bearer jwt_token_here"},
+ }
+ resp, err := client.Post("/app/installations/123/access_tokens", nil,
nil, headers)
+
+ assert.Nil(t, err)
+ assert.Equal(t, http.StatusCreated, resp.StatusCode)
+ assert.Equal(t, "POST", receivedMethod)
+ assert.Equal(t, "/app/installations/123/access_tokens", receivedPath)
+ assert.Equal(t, "Bearer jwt_token_here", receivedAuth)
+
+ body, _ := io.ReadAll(resp.Body)
+ resp.Body.Close()
+ assert.Contains(t, string(body), "ghs_test123")
+}
+
+func TestRefreshApiClientGet(t *testing.T) {
+ var receivedMethod string
+ var receivedQuery string
+
+ server := httptest.NewServer(http.HandlerFunc(func(w
http.ResponseWriter, r *http.Request) {
+ receivedMethod = r.Method
+ receivedQuery = r.URL.Query().Get("page")
+ w.WriteHeader(http.StatusOK)
+ w.Write([]byte(`{"ok":true}`))
+ }))
+ defer server.Close()
+
+ client := newRefreshApiClientWithTransport(server.URL,
server.Client().Transport)
+
+ resp, err := client.Get("/test", map[string][]string{"page": {"2"}},
nil)
+ assert.Nil(t, err)
+ assert.Equal(t, http.StatusOK, resp.StatusCode)
+ assert.Equal(t, "GET", receivedMethod)
+ assert.Equal(t, "2", receivedQuery)
+ resp.Body.Close()
+}
+
+func TestRefreshApiClientTimeout(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w
http.ResponseWriter, r *http.Request) {
+ // Sleep longer than the timeout
+ time.Sleep(2 * time.Second)
+ w.WriteHeader(http.StatusOK)
+ }))
+ defer server.Close()
+
+ // Use a client without its own timeout so the context timeout is the
only constraint
+ client := newRefreshApiClientWithTransport(server.URL,
http.DefaultTransport)
+ // Override the timeout to something short for the test
+ client.(*refreshApiClient).timeout = 100 * time.Millisecond
+
+ resp, err := client.Post("/slow", nil, nil, nil)
+ assert.NotNil(t, err)
+ if resp != nil {
+ resp.Body.Close()
+ }
+ // Verify the error is a deadline/context error
+ assert.True(t, strings.Contains(err.Error(), "deadline") ||
strings.Contains(err.Error(), "context"),
+ "expected deadline/context error, got: %s", err.Error())
+}
diff --git a/backend/plugins/github/token/round_tripper.go
b/backend/plugins/github/token/round_tripper.go
index 45ba3e9a7..8868572da 100644
--- a/backend/plugins/github/token/round_tripper.go
+++ b/backend/plugins/github/token/round_tripper.go
@@ -26,6 +26,11 @@ import (
// On 401's the round tripper will:
// - Force a refresh of the OAuth token via the TokenProvider
// - Retry the original request with the new token
+//
+// Note: When active, the RefreshRoundTripper overwrites the Authorization
header
+// on every request, superseding any header previously set by
SetupAuthentication.
+// This is intentional — for connections using token refresh (OAuth or GitHub
App),
+// the round tripper is the single source of truth for the current bearer
token.
type RefreshRoundTripper struct {
base http.RoundTripper
tokenProvider *TokenProvider
diff --git a/backend/plugins/github/token/round_tripper_test.go
b/backend/plugins/github/token/round_tripper_test.go
index e766adb88..2748d59e1 100644
--- a/backend/plugins/github/token/round_tripper_test.go
+++ b/backend/plugins/github/token/round_tripper_test.go
@@ -24,6 +24,7 @@ import (
"testing"
"time"
+ "github.com/apache/incubator-devlake/core/errors"
"github.com/apache/incubator-devlake/helpers/pluginhelper/api"
"github.com/apache/incubator-devlake/impls/logruslog"
"github.com/apache/incubator-devlake/plugins/github/models"
@@ -32,6 +33,36 @@ import (
"github.com/stretchr/testify/mock"
)
+// testRSAKey is a throwaway 2048-bit RSA private key used only in tests.
+// It is NOT a real credential.
+const testRSAKey = `-----BEGIN RSA PRIVATE KEY-----
+MIIEowIBAAKCAQEA1xJuX407giVTO/FY2pbp6bdB/XxaiPqAuvWIcEqabzq+d3ft
+O7fGtbXSQrCdtxEQt5dHFKdJofHcGlKPDnq1BNjWM3/xLWsQWQPSwUZ9H1qy/nDI
+GX+ciXmP8hbzoe5B1OXidAdrJUGWH3ox8Yp8OVd/yK9p34teCbzPnqVEc9tkgUT1
+94gHKLmvP28VefybFyGbYB3ujVIuA8Z26c4gQsyFzR2v1fVeDIu1e1afyH5WTFgr
+EWaztOo6pI5stzH00U7fNMzULBuYQ+ufQ4iQ7Ewt7fK5yyQNkx3pX0o1OQ/aYxQr
+hyBrHakxHBe78eq8rSR2KwSr5nDuYzIAOUhtnwIDAQABAoH/EknhZ6EkvRPR7G5e
+bq4/NduuSriDcITwbWuuuFPufzgdIZVzlu8xHfPOT9152qE2aD7XahGUk85fwbXh
+EeP64x/cA3FnmrAxPYfuUFnB0+i7KHGpzW2Wa/LqXlz3vS/UQePoDwl+xDYeUt2u
+xALHoSxZdlTWv6HLhLmw7ge9QJLc/xQO9dC678c4Y4JTCRrEvhE+eZiDEXb6HL2D
+uNMEwFqMLTxOurYKXE+iyzKg0e4D6oDkw6BC+vOUBnuH+9iKV6wnal24E1WnyhCa
+vy7iHBoc8npeKYAHjU5wKDQiWPMT9DjYBRucHvSyEYTeS5eyQHed42PGv+Ss2IL4
+w0JZAoGBAPjxiEJwck29NyfElw5+/P0IKIE8BkuIOSC8JoGfo+JpKYWFIXaLRbK8
+qvOiLDotHgC5IxFZyN8pejVe7Zvif5PepR9qMcV3e6Uz2nmJqtp5gFlzjNUSzbeM
+J6gqkejtwn7wJY3dwZhfbTTDDY7cZ9f3Cydarx+iu07unGCRT9DDAoGBAN0rG/F9
+tsOSwrgsnPwNeVPfJn2Zw6mvb1xuOoJl11tDnpS6tH50FBJ/TkERZak9+VZ761pq
+CvuMjLtePgC31rsAcUEEt1OwYCxne8gNxuzl2mWKD98ActbFf96VM4Q68Jvntxl1
+eOMMnSA+/yjvhpxdVabm3180mRu9eBv6SaH1AoGBAMpwL6xHoMwS6L1QIr7JCZYC
+gl3FoCDgIAS8vFuApFbDyd4oSvQJgZ49yo7g/DI66kEQTLIZXz4KjrTEA1lWsQRg
+c8q+Isc/yK6pIirfhq6vS25yhr3m0p9GPCGGrKzMW/O5+fAJuxrbzwSu8WGRXmjD
+HrDcD7kcLlGbvFLTGCLdAoGBALNID7W5Z16v6AItv++d6Hzxhh0IeRBi8s2lWO59
+KY6EiNcdZdSfuemoosGiHZuMbkMJ3qWDEnYI38e+xFoGrB0YZbYD4awIbF1yYWew
+q1E7ncbznJvznCO3I0lF/uWwdXyb39PWYvECN5h9GI+RYrf7/MN3oRhm5boT43oi
+cG/FAoGBALD1zSG9qrysLg8fw4dc4cs6dAHZhAszh47zw2WiMmUVWQZCbB+uhs44
+qGCAaer5KdnTqy9NdwW6rlcr4Y8jUFnMlMFA5HdmHkIfgTi+zi4Qf1mb9yzpJqnU
+jKflsh1Lyyqv2KsoIMz4vjew+lCVn80FZaEEQ1q9tAkyu5m53w65
+-----END RSA PRIVATE KEY-----`
+
func TestRoundTripper401Refresh(t *testing.T) {
mockRT := new(MockRoundTripper)
client := &http.Client{Transport: mockRT}
@@ -56,7 +87,7 @@ func TestRoundTripper401Refresh(t *testing.T) {
}
logger, _ := logruslog.NewDefaultLogger(logrus.New())
- tp := NewTokenProvider(conn, nil, client, logger)
+ tp := NewTokenProvider(conn, nil, client, logger, "")
rt := NewRefreshRoundTripper(mockRT, tp)
// Request
@@ -100,3 +131,163 @@ func TestRoundTripper401Refresh(t *testing.T) {
mockRT.AssertExpectations(t)
}
+
+func TestRoundTripper401WithAppKeyRefresh(t *testing.T) {
+ mockRT := new(MockRoundTripper)
+
+ expiry := time.Now().Add(10 * time.Minute) // Not expired (proactive
refresh won't trigger)
+ conn := &models.GithubConnection{
+ GithubConn: models.GithubConn{
+ GithubAccessToken: models.GithubAccessToken{
+ AccessToken: api.AccessToken{
+ Token: "old_app_token",
+ },
+ },
+ TokenExpiresAt: &expiry,
+ },
+ }
+ // Use tokens slice so GetToken returns the current token
+ conn.UpdateToken("old_app_token", "", &expiry, nil)
+
+ // refreshFn simulates minting a new installation token
+ refreshCalled := 0
+ tp := &TokenProvider{
+ conn: conn,
+ refreshFn: func(tp *TokenProvider) errors.Error {
+ refreshCalled++
+ newExpiry := time.Now().Add(1 * time.Hour)
+ tp.conn.UpdateToken("new_app_token", "", &newExpiry,
nil)
+ return nil
+ },
+ }
+
+ rt := NewRefreshRoundTripper(mockRT, tp)
+
+ req, _ := http.NewRequest("GET",
"https://api.github.com/repos/test/test";, nil)
+
+ // 1. First call returns 401
+ resp401 := &http.Response{
+ StatusCode: 401,
+ Body: io.NopCloser(bytes.NewBufferString("Bad
credentials")),
+ }
+ mockRT.On("RoundTrip", mock.MatchedBy(func(r *http.Request) bool {
+ return r.Header.Get("Authorization") == "Bearer old_app_token"
+ })).Return(resp401, nil).Once()
+
+ // 2. Retry call with new token (after refreshFn runs)
+ resp200 := &http.Response{
+ StatusCode: 200,
+ Body:
io.NopCloser(bytes.NewBufferString(`{"full_name":"test/test"}`)),
+ }
+ mockRT.On("RoundTrip", mock.MatchedBy(func(r *http.Request) bool {
+ return r.Header.Get("Authorization") == "Bearer new_app_token"
+ })).Return(resp200, nil).Once()
+
+ // Execute
+ resp, err := rt.RoundTrip(req)
+ assert.NoError(t, err)
+ assert.Equal(t, 200, resp.StatusCode)
+ assert.Equal(t, 1, refreshCalled, "refreshFn should have been called
exactly once")
+
+ body, _ := io.ReadAll(resp.Body)
+ assert.Equal(t, `{"full_name":"test/test"}`, string(body))
+
+ mockRT.AssertExpectations(t)
+}
+
+// TestProactiveRefreshNoDeadlock verifies that when the RefreshRoundTripper
wraps
+// the same http.Client's transport, a proactive token refresh does not
deadlock.
+// This reproduces the real-world scenario: GetToken() holds the mutex, calls
+// refreshGitHubAppInstallationToken, which makes an HTTP request. If that
request
+// goes through the RefreshRoundTripper (re-entering GetToken), it would
deadlock.
+// The fix is that the refresh uses baseTransport directly.
+func TestProactiveRefreshNoDeadlock(t *testing.T) {
+ // Set up a mock transport that will serve both:
+ // 1. The installation token refresh POST
+ // 2. The actual API GET after refresh
+ mockRT := new(MockRoundTripper)
+ client := &http.Client{Transport: mockRT}
+
+ // Token is expired — proactive refresh WILL trigger on GetToken()
+ expired := time.Now().Add(-1 * time.Minute)
+ conn := &models.GithubConnection{
+ GithubConn: models.GithubConn{
+ RestConnection: api.RestConnection{
+ Endpoint: "https://api.github.com/";,
+ },
+ MultiAuth: api.MultiAuth{
+ AuthMethod: models.AppKey,
+ },
+ GithubAccessToken: models.GithubAccessToken{
+ AccessToken: api.AccessToken{
+ Token: "expired_ghs_token",
+ },
+ },
+ GithubAppKey: models.GithubAppKey{
+ AppKey: api.AppKey{
+ AppId: "12345",
+ SecretKey: testRSAKey,
+ },
+ InstallationID: 99999,
+ },
+ TokenExpiresAt: &expired,
+ },
+ }
+ conn.UpdateToken("expired_ghs_token", "", &expired, nil)
+
+ // Create the TokenProvider with baseTransport = mockRT (the unwrapped
transport).
+ // This is what NewAppInstallationTokenProvider does: it captures
client.Transport
+ // BEFORE the caller wraps it with RefreshRoundTripper.
+ tp := NewAppInstallationTokenProvider(conn, nil, client, nil, "")
+
+ // Now wrap the client's transport with RefreshRoundTripper (simulating
what
+ // CreateApiClient does). After this, client.Transport =
RefreshRoundTripper,
+ // but tp.baseTransport still points to mockRT.
+ rt := NewRefreshRoundTripper(mockRT, tp)
+ client.Transport = rt
+
+ // Mock: installation token refresh POST (goes through baseTransport,
not RT)
+ newExpiry := time.Now().Add(1 * time.Hour)
+ installTokenBody := `{"token":"new_ghs_token","expires_at":"` +
newExpiry.Format(time.RFC3339) + `"}`
+ mockRT.On("RoundTrip", mock.MatchedBy(func(r *http.Request) bool {
+ return r.Method == "POST" && r.URL.Path ==
"/app/installations/99999/access_tokens"
+ })).Return(&http.Response{
+ StatusCode: 201,
+ Body:
io.NopCloser(bytes.NewBufferString(installTokenBody)),
+ }, nil).Once()
+
+ // Mock: the actual API request (goes through RefreshRoundTripper →
GetToken → mockRT)
+ mockRT.On("RoundTrip", mock.MatchedBy(func(r *http.Request) bool {
+ return r.Method == "GET" && r.URL.Path == "/repos/test/test"
+ })).Return(&http.Response{
+ StatusCode: 200,
+ Body:
io.NopCloser(bytes.NewBufferString(`{"full_name":"test/test"}`)),
+ }, nil).Once()
+
+ // Execute through the RefreshRoundTripper with a timeout to detect
deadlocks.
+ done := make(chan struct{})
+ var resp *http.Response
+ var reqErr error
+ go func() {
+ req, _ := http.NewRequest("GET",
"https://api.github.com/repos/test/test";, nil)
+ resp, reqErr = rt.RoundTrip(req)
+ close(done)
+ }()
+
+ select {
+ case <-done:
+ // Success — no deadlock
+ case <-time.After(5 * time.Second):
+ t.Fatal("DEADLOCK: RoundTrip did not complete within 5 seconds
— " +
+ "the refresh call is likely going through
RefreshRoundTripper instead of baseTransport")
+ }
+
+ assert.NoError(t, reqErr)
+ assert.Equal(t, 200, resp.StatusCode)
+ assert.Equal(t, "new_ghs_token", conn.Token, "token should have been
refreshed")
+
+ body, _ := io.ReadAll(resp.Body)
+ assert.Equal(t, `{"full_name":"test/test"}`, string(body))
+
+ mockRT.AssertExpectations(t)
+}
diff --git a/backend/plugins/github/token/token_provider.go
b/backend/plugins/github/token/token_provider.go
index ed3fa2256..6af31aaf5 100644
--- a/backend/plugins/github/token/token_provider.go
+++ b/backend/plugins/github/token/token_provider.go
@@ -40,23 +40,57 @@ const (
)
type TokenProvider struct {
- conn *models.GithubConnection
- dal dal.Dal
- httpClient *http.Client
- logger log.Logger
- mu sync.Mutex
- refreshURL string
+ conn *models.GithubConnection
+ dal dal.Dal
+ encryptionSecret string
+ httpClient *http.Client
+ baseTransport http.RoundTripper // original transport, before
RefreshRoundTripper wrapping
+ logger log.Logger
+ mu sync.Mutex
+ refreshURL string
+ refreshFn func(*TokenProvider) errors.Error
}
// NewTokenProvider creates a TokenProvider for the given GitHub connection
using
// the provided DAL, HTTP client, and logger, and returns a pointer to it.
-func NewTokenProvider(conn *models.GithubConnection, d dal.Dal, client
*http.Client, logger log.Logger) *TokenProvider {
+func NewTokenProvider(conn *models.GithubConnection, d dal.Dal, client
*http.Client, logger log.Logger, encryptionSecret string) *TokenProvider {
return &TokenProvider{
- conn: conn,
- dal: d,
- httpClient: client,
- logger: logger,
- refreshURL: "https://github.com/login/oauth/access_token";,
+ conn: conn,
+ dal: d,
+ encryptionSecret: encryptionSecret,
+ httpClient: client,
+ logger: logger,
+ refreshURL: "https://github.com/login/oauth/access_token";,
+ }
+}
+
+// NewAppInstallationTokenProvider creates a TokenProvider that refreshes
GitHub App installation tokens.
+// IMPORTANT: Call this BEFORE wrapping the client's transport with
RefreshRoundTripper,
+// so that baseTransport captures the unwrapped transport and refresh calls
don't deadlock.
+func NewAppInstallationTokenProvider(conn *models.GithubConnection, d dal.Dal,
client *http.Client, logger log.Logger, encryptionSecret string) *TokenProvider
{
+ if logger != nil {
+ expiresStr := "unknown"
+ if conn.TokenExpiresAt != nil {
+ expiresStr = conn.TokenExpiresAt.Format(time.RFC3339)
+ }
+ logger.Info("Created AppInstallation token provider for
connection %d (installation %d, token expires at %s)",
+ conn.ID, conn.InstallationID, expiresStr)
+ }
+ // Capture the transport now, before the caller wraps it with
RefreshRoundTripper.
+ // This avoids a deadlock: refresh calls must bypass the
RefreshRoundTripper that
+ // holds the TokenProvider mutex during GetToken().
+ baseTransport := client.Transport
+ if baseTransport == nil {
+ baseTransport = http.DefaultTransport
+ }
+ return &TokenProvider{
+ conn: conn,
+ dal: d,
+ encryptionSecret: encryptionSecret,
+ httpClient: client,
+ baseTransport: baseTransport,
+ logger: logger,
+ refreshFn: refreshGitHubAppInstallationToken,
}
}
@@ -65,6 +99,14 @@ func (tp *TokenProvider) GetToken() (string, errors.Error) {
defer tp.mu.Unlock()
if tp.needsRefresh() {
+ if tp.logger != nil {
+ expiresStr := "unknown"
+ if tp.conn.TokenExpiresAt != nil {
+ expiresStr =
tp.conn.TokenExpiresAt.Format(time.RFC3339)
+ }
+ tp.logger.Info("Proactive token refresh triggered for
connection %d (token expires at %s)",
+ tp.conn.ID, expiresStr)
+ }
if err := tp.refreshToken(); err != nil {
return "", err
}
@@ -73,10 +115,6 @@ func (tp *TokenProvider) GetToken() (string, errors.Error) {
}
func (tp *TokenProvider) needsRefresh() bool {
- if tp.conn.RefreshToken == "" {
- return false
- }
-
buffer := DefaultRefreshBuffer
if envBuffer := os.Getenv("GITHUB_TOKEN_REFRESH_BUFFER_MINUTES");
envBuffer != "" {
if val, err := strconv.Atoi(envBuffer); err == nil {
@@ -84,6 +122,16 @@ func (tp *TokenProvider) needsRefresh() bool {
}
}
+ if tp.refreshFn != nil {
+ if tp.conn.TokenExpiresAt == nil {
+ return false
+ }
+ return time.Now().Add(buffer).After(*tp.conn.TokenExpiresAt)
+ }
+
+ if tp.conn.RefreshToken == "" {
+ return false
+ }
if tp.conn.TokenExpiresAt == nil {
return false
}
@@ -91,6 +139,9 @@ func (tp *TokenProvider) needsRefresh() bool {
}
func (tp *TokenProvider) refreshToken() errors.Error {
+ if tp.refreshFn != nil {
+ return tp.refreshFn(tp)
+ }
tp.logger.Info("Refreshing GitHub token for connection %d", tp.conn.ID)
data := map[string]string{
@@ -159,13 +210,12 @@ func (tp *TokenProvider) refreshToken() errors.Error {
)
if tp.dal != nil {
- err := tp.dal.UpdateColumns(tp.conn, []dal.DalSet{
- {ColumnName: "token", Value: tp.conn.Token},
- {ColumnName: "refresh_token", Value:
tp.conn.RefreshToken},
- {ColumnName: "token_expires_at", Value:
tp.conn.TokenExpiresAt},
- {ColumnName: "refresh_token_expires_at", Value:
tp.conn.RefreshTokenExpiresAt},
- })
- if err != nil {
+ // Manually encrypt and use UpdateColumns to persist only the
token-related
+ // columns. We cannot use dal.Update (GORM Save) because it
writes ALL fields
+ // including refresh_token_expires_at which may have Go zero
time that MySQL
+ // rejects. We cannot use UpdateColumns with plaintext because
it bypasses the
+ // GORM encdec serializer. So we encrypt manually and write the
ciphertext.
+ if err := PersistEncryptedTokenColumns(tp.dal, tp.conn,
tp.encryptionSecret, tp.logger, true); err != nil {
tp.logger.Warn(err, "failed to persist refreshed token")
}
}
@@ -184,8 +234,14 @@ func (tp *TokenProvider) ForceRefresh(oldToken string)
errors.Error {
// If the token has changed since the request was made, it means
another thread
// has already refreshed it.
if tp.conn.Token != oldToken {
+ if tp.logger != nil {
+ tp.logger.Info("Skipping reactive token refresh for
connection %d — token already changed by another goroutine", tp.conn.ID)
+ }
return nil
}
+ if tp.logger != nil {
+ tp.logger.Info("Reactive token refresh triggered for connection
%d (received 401)", tp.conn.ID)
+ }
return tp.refreshToken()
}
diff --git a/backend/plugins/github/token/token_provider_test.go
b/backend/plugins/github/token/token_provider_test.go
index 3319f5591..a754ad63c 100644
--- a/backend/plugins/github/token/token_provider_test.go
+++ b/backend/plugins/github/token/token_provider_test.go
@@ -29,7 +29,6 @@ import (
"github.com/apache/incubator-devlake/core/errors"
"github.com/apache/incubator-devlake/helpers/pluginhelper/api"
"github.com/apache/incubator-devlake/impls/logruslog"
- mockdal "github.com/apache/incubator-devlake/mocks/core/dal"
"github.com/apache/incubator-devlake/plugins/github/models"
"github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert"
@@ -93,7 +92,7 @@ func TestTokenProviderConcurrency(t *testing.T) {
}
logger, _ := logruslog.NewDefaultLogger(logrus.New())
- tp := NewTokenProvider(conn, nil, client, logger)
+ tp := NewTokenProvider(conn, nil, client, logger, "")
// Mock response for refresh
respBody :=
`{"access_token":"new_token","refresh_token":"new_refresh_token","expires_in":3600,"refresh_token_expires_in":3600}`
@@ -143,44 +142,108 @@ func TestConfigurableBuffer(t *testing.T) {
assert.False(t, tp.needsRefresh())
}
-func TestPersistenceFailure(t *testing.T) {
- mockRT := new(MockRoundTripper)
- client := &http.Client{Transport: mockRT}
- mockDal := new(mockdal.Dal)
+// fakeRefreshFn returns a refreshFn that updates the connection token to
newToken
+// and increments the call counter pointed to by count.
+func fakeRefreshFn(newToken string, count *int) func(*TokenProvider)
errors.Error {
+ return func(tp *TokenProvider) errors.Error {
+ *count++
+ newExpiry := time.Now().Add(1 * time.Hour)
+ tp.conn.UpdateToken(newToken, "", &newExpiry, nil)
+ return nil
+ }
+}
+
+func TestNeedsRefreshWithRefreshFn(t *testing.T) {
+ callCount := 0
+ tp := &TokenProvider{
+ conn: &models.GithubConnection{},
+ refreshFn: fakeRefreshFn("unused", &callCount),
+ }
+
+ // Token not expired — outside default 5m buffer
+ expiry1 := time.Now().Add(10 * time.Minute)
+ tp.conn.TokenExpiresAt = &expiry1
+ assert.False(t, tp.needsRefresh(), "should not refresh when token is
10m from expiry")
+
+ // Token inside 5m buffer
+ expiry2 := time.Now().Add(2 * time.Minute)
+ tp.conn.TokenExpiresAt = &expiry2
+ assert.True(t, tp.needsRefresh(), "should refresh when token is 2m from
expiry")
+ // Token already expired
+ expiry3 := time.Now().Add(-1 * time.Minute)
+ tp.conn.TokenExpiresAt = &expiry3
+ assert.True(t, tp.needsRefresh(), "should refresh when token is
expired")
+
+ // TokenExpiresAt is nil — can't determine expiry, don't refresh (401
fallback covers this)
+ tp.conn.TokenExpiresAt = nil
+ assert.False(t, tp.needsRefresh(), "should not refresh when
TokenExpiresAt is nil")
+
+ // Provider with neither refreshFn nor RefreshToken — should never
refresh
+ tp2 := &TokenProvider{
+ conn: &models.GithubConnection{},
+ }
+ expiry4 := time.Now().Add(-1 * time.Minute)
+ tp2.conn.TokenExpiresAt = &expiry4
+ assert.False(t, tp2.needsRefresh(), "should not refresh without
refreshFn or RefreshToken")
+}
+
+func TestAppKeyGetTokenTriggersRefresh(t *testing.T) {
+ callCount := 0
+ expired := time.Now().Add(-1 * time.Minute)
conn := &models.GithubConnection{
GithubConn: models.GithubConn{
- RefreshToken: "refresh_token",
GithubAccessToken: models.GithubAccessToken{
AccessToken: api.AccessToken{
- Token: "old_token",
+ Token: "expired_token",
},
},
- GithubAppKey: models.GithubAppKey{
- AppKey: api.AppKey{
- AppId: "123",
- SecretKey: "secret",
+ TokenExpiresAt: &expired,
+ },
+ }
+
+ tp := &TokenProvider{
+ conn: conn,
+ refreshFn: fakeRefreshFn("refreshed_app_token", &callCount),
+ }
+
+ token, err := tp.GetToken()
+ assert.NoError(t, err)
+ assert.Equal(t, "refreshed_app_token", token)
+ assert.Equal(t, 1, callCount, "refreshFn should have been called
exactly once")
+
+ // Second call — token is now fresh, should not trigger refresh
+ token2, err := tp.GetToken()
+ assert.NoError(t, err)
+ assert.Equal(t, "refreshed_app_token", token2)
+ assert.Equal(t, 1, callCount, "refreshFn should not be called again for
a fresh token")
+}
+
+func TestAppKeyForceRefresh(t *testing.T) {
+ callCount := 0
+ conn := &models.GithubConnection{
+ GithubConn: models.GithubConn{
+ GithubAccessToken: models.GithubAccessToken{
+ AccessToken: api.AccessToken{
+ Token: "old_app_token",
},
},
},
}
- logger, _ := logruslog.NewDefaultLogger(logrus.New())
- tp := NewTokenProvider(conn, mockDal, client, logger)
-
- // Mock response for refresh
- respBody :=
`{"access_token":"new_token","refresh_token":"new_refresh_token","expires_in":3600,"refresh_token_expires_in":3600}`
- resp := &http.Response{
- StatusCode: 200,
- Body: io.NopCloser(bytes.NewBufferString(respBody)),
+ tp := &TokenProvider{
+ conn: conn,
+ refreshFn: fakeRefreshFn("new_app_token", &callCount),
}
- mockRT.On("RoundTrip", mock.Anything).Return(resp, nil).Once()
- // Mock DAL failure
- mockDal.On("UpdateColumns", mock.Anything, mock.Anything,
mock.AnythingOfType("[]dal.Clause")).Return(errors.Default.New("db error"))
- err := tp.ForceRefresh("old_token")
- assert.NoError(t, err) // Should not return error even if persistence
fails
+ // ForceRefresh with matching old token — should trigger refresh
+ err := tp.ForceRefresh("old_app_token")
+ assert.NoError(t, err)
+ assert.Equal(t, 1, callCount)
+ assert.Equal(t, "new_app_token", conn.Token)
- mockRT.AssertExpectations(t)
- mockDal.AssertExpectations(t)
+ // ForceRefresh with stale old token — token has already changed,
should be a no-op
+ err = tp.ForceRefresh("old_app_token")
+ assert.NoError(t, err)
+ assert.Equal(t, 1, callCount, "should not refresh when token has
already changed")
}
