Hi, We are seeing intermittent occurrences of 502 Bad Gateway error in our server. About 0.5% requests fail with this error.
Out setup is: Flex running jetty9-compat F1 machine 1 server Our request pattern is bursty. So the server gets ~30 requests in parallel. The failures, when they happen are clustered, that is over a period of 10'ish seconds one would see 3-4 errors. The requests which complete successfully, finish in 50-100 ms, so it does not appear like the server is under major load and not able to keep up. To rule out this possibility, I started the servers with 5 replicas. However the failure percentage did not change. >From the looks of it, it appears that there is some throttling or quota issue at play. I tried tweaking max-concurrent-requests param. Set it to 300, but that did not make any difference either. I do not see new instances being created at the time of failure either. The request log for the failed request: 09:57:30.686POST502262 B4 msAppEngine-Google; (+ http://code.google.com/appengine; appid: s~village-test)/read 107.178.194.3 - - [07/Feb/2017:09:57:30 +0530] "POST /read HTTP/1.1" 502 262 - "AppEngine-Google; (+http://code.google.com/appengine; ms=4 cpu_ms=0 cpm_usd=2.9279999999999998e-8 loading_request=0 instance=- app_engine_release=1.9.48 trace_id=- { protoPayload: {…} insertId: "58994cb30002335cb47fd364" httpRequest: {…} resource: {…} timestamp: "2017-02-07T04:27:30.686052Z" labels: {…} operation: {…} } Looking around at other logs at around the time of failure I see. 09:57:30.000[error] 32#32: *35107 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 169.254.160.2, server: , request: "POST /read HTTP/1.1", upstream: " http://172.17.0.4:8080/read";, host: "bigtable-dev.appspot.com" AFAICT this request never made it to our servlet. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/2f5a728b-38ff-4de7-b260-32c1e15f5351%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
