RubenRickardSuncorp opened a new issue, #5349: URL: https://github.com/apache/incubator-devlake/issues/5349
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened When adding a new connector via the config ui the the browser get's a bad gateway error. Upon investigation the devlake backend had actually crashed. This happens consistently. I've tried Jenkins, Jira and Webhook connections. It happens when testing the connection or saving it. It also happens when creating a connection directly via API call. **devlake-config-ui logs** ` 10.103.150.114 - - [01/Jun/2023:23:59:33 +0000] "GET /api/version HTTP/1.1" 200 22 "https://devlake-a-000994.rosa-nonp-priv.os.app.int.corp.sun/"; "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "10.76.128.50" 10.103.150.114 - - [01/Jun/2023:23:59:33 +0000] "GET /api/projects?page=1&pageSize=200 HTTP/1.1" 200 276 "https://devlake-a-000994.rosa-nonp-priv.os.app.int.corp.sun/projects"; "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "10.76.128.50" 10.103.150.114 - - [01/Jun/2023:23:59:36 +0000] "GET /api/plugins/webhook/connections HTTP/1.1" 200 2 "https://devlake-a-000994.rosa-nonp-priv.os.app.int.corp.sun/connections/webhook"; "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "10.76.128.50" 2023/06/01 23:59:36 [error] 16#16: *5 upstream prematurely closed connection while reading response header from upstream, client: 10.103.150.114, server: localhost, request: "GET /api/plugins/webhook/connections HTTP/1.1", upstream: "http://10.102.120.21:8080/plugins/webhook/connections";, host: "devlake-a-000994.rosa-nonp-priv.os.app.int.corp.sun", referrer: "https://devlake-a-000994.rosa-nonp-priv.os.app.int.corp.sun/connections/webhook"; 10.103.150.114 - - [01/Jun/2023:23:59:36 +0000] "GET /api/plugins/webhook/connections HTTP/1.1" 502 559 "https://devlake-a-000994.rosa-nonp-priv.os.app.int.corp.sun/connections/webhook"; "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "10.76.128.50" 2023/06/01 23:59:41 [error] 17#17: *11 connect() failed (111: Connection refused) while connecting to upstream, client: 10.103.150.114, server: localhost, request: "POST /api/plugins/webhook/connections HTTP/1.1", upstream: "http://10.102.120.21:8080/plugins/webhook/connections";, host: "devlake-a-000994.rosa-nonp-priv.os.app.int.corp.sun", referrer: "https://devlake-a-000994.rosa-nonp-priv.os.app.int.corp.sun/connections/webhook"; 10.103.150.114 - - [01/Jun/2023:23:59:41 +0000] "POST /api/plugins/webhook/connections HTTP/1.1" 502 559 "https://devlake-a-000994.rosa-nonp-priv.os.app.int.corp.sun/connections/webhook"; "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "10.76.128.50" ` **devlake logs in debug mode** ` [GIN-debug] Listening and serving HTTP on 0.0.0.0:8080 [GIN] 2023/06/01 - 23:59:33 | 200 | 220.199µs | 10.76.128.50 | GET "/version" [GIN] 2023/06/01 - 23:59:33 | 200 | 37.024059ms | 10.76.128.50 | GET "/projects?page=1&pageSize=200" [GIN] 2023/06/01 - 23:59:36 | 200 | 24.973856ms | 10.76.128.50 | GET "/plugins/webhook/connections" runtime: g 86: unexpected return pc for net/http.serverHandler.ServeHTTP called from 0x17920 stack: frame={sp:0xc000465ac8, fp:0xc000465b78} stack=[0xc000464000,0xc000466000) 0x000000c0004659c8: 0x00000000000000e0 0x000000c000540b40 0x000000c0004659d8: 0x0000000000000005 0x0000000000000005 0x000000c0004659e8: 0x0000000000000000 0x0000000001b69100 0x000000c0004659f8: 0x000000c00004ede0 0x000000000000001c 0x000000c000465a08: 0x0000000000000000 0x0000000000000000 0x000000c000465a18: 0x0000000000000000 0x00000001003b5200 0x000000c000465a28: 0x0000000000000000 0x0000000000a45b3e <sync.(*Pool).pin+0x000000000000001e> 0x000000c000465a38: 0x0000000000add429 <strings.Index+0x0000000000000169> 0x000000c000540b40 0x000000c000465a48: 0x0000000000000005 0x0000000000000005 0x000000c000465a58: 0x0000000000000000 0x0000000001b69100 0x000000c000465a68: 0x000000c00004ede0 0x000000000000001c 0x000000c000465a78: 0x000000c000465ab8 0x000000000133ce52 <github.com/gin-gonic/gin.(*Engine).ServeHTTP+0x00000000000001d2> 0x000000c000465a88: 0x000000c000103a00 0x000000c00058ab00 0x000000c000465a98: 0x0000000000000000 0x000000c00058ab00 0x000000c000465aa8: 0x000000c000103b18 0x0000000001e091a0 0x000000c000465ab8: 0x000000c000465b68 0x0000000000e6264c <net/http.serverHandler.ServeHTTP+0x000000000000030c> 0x000000c000465ac8: <0x000000c000103a00 0x0000000001e091a0 0x000000c000465ad8: 0x000000c0000f3180 0x000000c00058af00 0x000000c000465ae8: 0x0000000000a11b71 <runtime.newproc+0x0000000000000051> 0x000000c00008faf8 0x000000c000465af8: 0x0000000000a11ba0 <runtime.newproc.func1+0x0000000000000000> 0x000000c00008fb18 0x000000c000465b08: 0x0000000001df9fc0 0x0000000000e56caa <net/http.(*connReader).startBackgroundRead+0x00000000000000ca> 0x000000c000465b18: 0x0000000000a38416 <runtime.asmcgocall+0x0000000000000036> 0x00000000000177c1 0x000000c000465b28: 0x000000c000103a00 0x000000c00008fb40 0x000000c000465b38: 0x000000c001f504b0 0x0000000000017bd0 0x000000c000465b48: 0x000000c0000f8000 0x0000000000017b77 0x000000c000465b58: 0x0000000000a383e0 <runtime.asmcgocall+0x0000000000000000> 0x0000000000000000 0x000000c000465b68: 0x000000c00008ffa8 !0x0000000000017920 0x000000c000465b78: >0x000000c0003faf90 0x0000000001e091a0 0x000000c000465b88: 0x000000c0000f3180 0x000000c00058af00 0x000000c000465b98: 0x0000000001aab901 0x000000c0003faf00 0x000000c000465ba8: 0x0000000000000000 0x0000000000000000 0x000000c000465bb8: 0x0000000000000000 0x0000000000000000 0x000000c000465bc8: 0x0000000000000000 0x0000000000000000 0x000000c000465bd8: 0x0000000000000000 0x0000000000000000 0x000000c000465be8: 0x0000000000000000 0x0000000000000000 0x000000c000465bf8: 0x0000000000000000 0x0000000000000000 0x000000c000465c08: 0x0000000000000000 0x0000000000000000 0x000000c000465c18: 0x0000000000000000 0x0000000000000000 0x000000c000465c28: 0x0000000000000000 0x0000000000000000 0x000000c000465c38: 0x0000000000000000 0x0000000000000000 0x000000c000465c48: 0x0000000000000000 0x0000000000000000 0x000000c000465c58: 0x0000000000000000 0x0000000000000000 0x000000c000465c68: 0x0000000000000000 0x0000000000000000 fatal error: unknown caller pc ` ### What do you expect to happen A new connection would be successfully created. ### How to reproduce Create any new connector ### Anything else The only similar crash logs I could find online where related to Dynatrace. I have added an annotation to the pod to prevent Dynatrace injecting one-agent. I had to disabled remote plugins due to #5128. ### Version Reproduced on v0.16.0 and v0.17.0-beta8 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
