pballester opened a new issue, #9021:
URL: https://github.com/apache/devlake/issues/9021
## What happened
Every `plugins/org` API endpoint (`teams.csv`, `users.csv`,
`user_account_mapping.csv`, GET and PUT) returns HTTP 500 with a nil-pointer
panic. Verified on v1.0.2 and v1.0.3-beta15 (helm deployment, MySQL backend).
GET with `?fake_data=true` works (it never touches the store).
Panic (v1.0.3-beta15, `PUT /plugins/org/teams.csv`):
```
runtime error: invalid memory address or nil pointer dereference
/app/plugins/org/api/team.go:87 <- h.store.deleteAll(&crossdomain.Team{})
/app/server/api/router.go:149
```
GET without fake_data panics at `team.go:47` (`h.store.findAllTeams()`).
## Root cause (with evidence)
It's an initialization-order race between route registration and plugin
initialization:
1. `runner.LoadGoPlugins` only does `Lookup("PluginEntry")` +
`plugin.RegisterPlugin(...)` — it does **not** call `PluginInit.Init()`
([loader.go](https://github.com/apache/devlake/blob/v1.0.3-beta15/backend/core/runner/loader.go#L62-L83)).
2. `plugin.InitPlugins(basicRes)` — the only place `Init()` runs in server
mode — is invoked from `pipelineServiceInit()`
([pipeline.go:83](https://github.com/apache/devlake/blob/v1.0.3-beta15/backend/server/services/pipeline.go#L83)).
3. `registerPluginEndpoints` builds the router from each plugin's
`ApiResources()`. For `org`, that returns **bound method values** on instance
state: `p.handlers.GetTeam`, etc.
([impl.go](https://github.com/apache/devlake/blob/v1.0.3-beta15/backend/plugins/org/impl/impl.go#L104-L120)).
4. In Go, a method value captures its receiver **at evaluation time**. If
`ApiResources()` is evaluated before `Init()` set `p.handlers`, the router
permanently holds handlers bound to a nil `*Handlers` — running `Init()` later
doesn't rebind them.
Startup timeline from our pod logs proving the order (a DB migration was
pending at boot, which delays `pipelineServiceInit` while the HTTP server must
start to serve `/proceed-db-migration`):
```
12:13:58 plugin loaded org
12:14:11 endpoint PUT /plugins/org/teams.csv ... registerPluginEndpoints
<- ApiResources() evaluated, p.handlers == nil
12:14:23 plugin: <X> doesn't implement 'PluginInit', it will be skipped.
<- InitPlugins runs, 12s too late
```
`org` is the only plugin whose `ApiResources()` binds per-instance state
created in `Init()` — other plugins read package-level vars at call time —
which is why only `org` breaks, and why it breaks non-deterministically across
deployments (whoever wins the race). This likely explains the long trail of
closed-as-stale reports: #8590, #8271, #7957, #7219 (different symptoms of the
same race — reporters couldn't pin it because it depends on startup timing; a
pending DB migration at boot makes it deterministic).
Note the v1.0.3-beta fix `(&team{}).toDomainLayer(tt)` (team.go:84) resolved
a *second, independent* nil deref, which made the remaining one harder to spot.
## What do you expect to happen
`plugins/org` CSV endpoints work regardless of startup timing.
## How to reproduce
1. Deploy v1.0.2 or v1.0.3-beta15 (helm chart, MySQL) with a pending DB
migration at boot (e.g. any version upgrade), or any startup where route
registration beats `pipelineServiceInit`.
2. Confirm the migration, then `curl -X PUT <endpoint>/plugins/org/teams.csv
-F '[email protected]'` → HTTP 500, panic above. `GET /plugins/org/teams.csv` →
same at team.go:47.
## Proposed fix (happy to submit a PR)
Two options, smallest first:
**A (org-scoped, minimal):** make `org`'s `ApiResources()` resolve handlers
at call time instead of bind time — closures over `p` (pointer receiver) rather
than method values over `p.handlers`:
```go
func (p *Org) ApiResources() map[string]map[string]plugin.ApiResourceHandler
{
wrap := func(f func(*api.Handlers) plugin.ApiResourceHandler)
plugin.ApiResourceHandler {
return func(in *plugin.ApiResourceInput) (*plugin.ApiResourceOutput,
errors.Error) {
if p.handlers == nil {
return nil, errors.Internal.New("org plugin not initialized
yet")
}
return f(p.handlers)(in)
}
}
...
}
```
**B (framework):** guarantee `plugin.InitPlugins()` runs before
`registerPluginEndpoints` (e.g. right after `runner.LoadPlugins` in
`services.Init`), decoupling it from `pipelineServiceInit`. Fixes the class of
bug for any future plugin with instance state, but touches startup ordering for
everything.
## Version
v1.0.3-beta15 (also reproduced on v1.0.2). Helm chart on Kubernetes (EKS),
external MySQL 8.0.
## Are you willing to submit a PR?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
