Any reason 408 would be undesirable? https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/408
On Thu, Apr 18, 2019 at 10:37 AM Robert Newson <rnew...@apache.org> wrote: > 503 imo. > > -- > Robert Samuel Newson > rnew...@apache.org > > On Thu, 18 Apr 2019, at 18:24, Adam Kocoloski wrote: > > Yes, we should. Currently it’s a 500, maybe there’s something more > appropriate: > > > > > https://github.com/apache/couchdb/blob/8ef42f7241f8788afc1b6e7255ce78ce5d5ea5c3/src/chttpd/src/chttpd.erl#L947-L949 > > > > Adam > > > > > On Apr 18, 2019, at 12:50 PM, Joan Touzet <woh...@apache.org> wrote: > > > > > > What happens when it turns out the client *hasn't* timed out and we > > > just...hang up on them? Should we consider at least trying to send back > > > some sort of HTTP status code? > > > > > > -Joan > > > > > > On 2019-04-18 10:58, Garren Smith wrote: > > >> I'm +1 on this. With partition queries, we added a few more timeouts > that > > >> can be enabled which Cloudant enable. So having the ability to shed > old > > >> requests when these timeouts get hit would be great. > > >> > > >> Cheers > > >> Garren > > >> > > >> On Tue, Apr 16, 2019 at 2:41 AM Adam Kocoloski <kocol...@apache.org> > wrote: > > >> > > >>> Hi all, > > >>> > > >>> For once, I’m coming to you with a topic that is not strictly about > > >>> FoundationDB :) > > >>> > > >>> CouchDB offers a few config settings (some of them undocumented) to > put a > > >>> limit on how long the server is allowed to take to generate a > response. The > > >>> trouble with many of these timeouts is that, when they fire, they do > not > > >>> actually clean up all of the work that they initiated. A couple of > examples: > > >>> > > >>> - Each HTTP response coordinated by the “fabric” application spawns > > >>> several ephemeral processes via “rexi" on different nodes in the > cluster to > > >>> retrieve data and send it back to the process coordinating the > response. If > > >>> the request timeout fires, the coordinating process will be killed > off, but > > >>> the ephemeral workers might not be. In a healthy cluster they’ll > exit on > > >>> their own when they finish their jobs, but there are conditions > under which > > >>> they can sit around for extended periods of time waiting for an > overloaded > > >>> gen_server (e.g. couch_server) to respond. > > >>> > > >>> - Those named gen_servers (like couch_server) responsible for > serializing > > >>> access to important data structures will dutifully process messages > > >>> received from old requests without any regard for (of even knowledge > of) > > >>> the fact that the client that sent the message timed out long ago. > This can > > >>> lead to a sort of death spiral in which the gen_server is ultimately > > >>> spending ~all of its time serving dead clients and every client is > timing > > >>> out. > > >>> > > >>> I’d like to see us introduce a documented maximum request duration > for all > > >>> requests except the _changes feed, and then use that information to > aid in > > >>> load shedding throughout the stack. We can audit the codebase for > > >>> gen_server calls with long timeouts (I know of a few on the critical > path > > >>> that set their timeouts to `infinity`) and we can design servers that > > >>> efficiently drop old requests, knowing that the client who made the > request > > >>> must have timed out. A couple of topics for discussion: > > >>> > > >>> - the “gen_server that sheds old requests” is a very generic > pattern, one > > >>> that seems like it could be well-suited to its own behaviour. A > cursory > > >>> search of the internet didn’t turn up any prior art here, which > surprises > > >>> me a bit. I’m wondering if this is worth bringing up with the broader > > >>> Erlang community. > > >>> > > >>> - setting and enforcing timeouts is a healthy pattern for read-only > > >>> requests as it gives a lot more feedback to clients about the health > of the > > >>> server. When it comes to updates things are a little bit more muddy, > just > > >>> because there remains a chance that an update can be committed, but > the > > >>> caller times out before learning of the successful commit. We should > try to > > >>> minimize the likelihood of that occurring. > > >>> > > >>> Cheers, Adam > > >>> > > >>> P.S. I did say that this wasn’t _strictly_ about FoundationDB, but of > > >>> course FDB has a hard 5 second limit on all transactions, so it is a > bit of > > >>> a forcing function :).Even putting FoundationDB aside, I would still > argue > > >>> to pursue this path based on our Ops experience with the current > codebase. > > >> > > > > > > > >
