For me, I think Bob's approach below, combined with ensuring the overhead of coding the endpoint is low by setting up a combined metrics source that both Prometheus and the JSON-based metrics endpoint would use feels cleanest -- one internal API and formatting based on the endpoint called. Based on Will's comments, this would need to do the work to format things sensibly for Prometheus and also be tightly coupled enough to CouchDB to process the underlying histograms into Prometheus consumable ones.
I think this maintains CouchDB's "JSON by default" stance neatly while avoiding what feel like undesirable ways to specify the content type to workaround Prometheus client or spec weaknesses. I also think that this works well in terms of future proofing -- for me it's cleaner to remove an entire (optionally enabled) endpoint than an option on an existing endpoint. Hopefully it is also easier from a client library perspective; is it? -- Mike. On Wed, 23 Sep 2020, at 21:43, Robert Samuel Newson wrote: > Hi, > > I don't see why this can't be a new endpoint (emitting the normal > Prometheus format) that couchdb administrators can choose to enable > (and leave it disabled by default, returning a 404). > > I agree with the general view that content type negotiation doesn't > really work well in practice, and I don't much like the suggested > ?accept= hack. > > I am old and world-weary and have seen these sorts of things come and > go many times. Prometheus seems a fine option for now, and perhaps for > a while, but it feels like a plugin, not core, to me. > > B. > > > On 23 Sep 2020, at 17:25, Richard Ellis <ricel...@uk.ibm.com> wrote: > > > >> so we should absolutely make this info available in JSON > > > > This sounds like a good idea to me > > > >> we could fall back to a ?accept=prometheus option > > > > I'm opposed to adding endpoints that supply different content-type > > responses via non-standard means. The CouchDB API has some examples of > > this through history and it can make using those endpoints with standard > > tooling somewhat painful. > > > > A bit of quick searching seems to suggest that the format has its own > > project https://openmetrics.io/ - and this declares it's text > > representation linking back to > > https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format > > > > which declares a Content-Type of "text/plain; version=0.0.4" - so > > defaulting to that, but following Joan's suggestion and switching to JSON > > for a supplied Accept:application/json in the standard way seems a like > > good choice to me. > > > > Rich > > > > > > > > From: Jan Lehnardt <j...@apache.org> > > To: dev@couchdb.apache.org > > Cc: "Gesellchen, Tobias" <tobias.gesellc...@europace.de> > > Date: 23/09/2020 16:42 > > Subject: [EXTERNAL] Re: [DISCUSS] Prometheus endpoint in CouchDB > > 4.x > > > > > > > > Hi all, > > > > a few things to consider: > > > > 1. The idea of unifying our “get runtime info about CouchDB” endpoints > > into one is solid, as it is always weird to make sure you know which info > > you get where. We see this specifically in support engagements, where it > > is always awkward to ask for the results of multiple endpoints. > > > > 2. This directly leads to the question about what the endpoint should be > > called. I feel if it is a new endpoint, we should give it a new name. > > _info maybe, but feel free to bike shed away. > > > > 3. Next the question about per-node and per-cluster info/metrics/activity > > on the endpoint. It might be convenient to be able to ask any one node > > about what is going on in the entire cluster, rather than any one node, > > but some stats only make sense in the context of a single node. Maybe the > > result includes everything separated by node somehow. > > > > 4. Then the format: if this wasn’t about Prometheus and its custom format, > > we wouldn’t discuss any of this and just use JSON. Since we *do* want to > > target Prometheus with this, we have to talk about the format. Any of the > > above is useful for non-Prometheus consumers, so we should absolutely make > > this info available in JSON. And we can *also* send it in the Prometheus > > format. The “correct” HTTP-way of doing this would be to use the Accept > > header on the new endpoint, as Joan points out, but that’s often not an > > option, so we could fall back to a ?accept=prometheus option. This would > > also leave us open to add more formats in the future, as new standards > > arise. > > > > 5. That leads us to whether we want to do this. Every five or so years, > > new standards for these types of systems arise, and sometimes it is worth > > incorporating them (like we finally do with the SystemD compatible log > > formatter) and sometimes it is not and folks write tools to convert from > > our HTTP/JSON standard to whatever they need ( > > https://github.com/gesellix/couchdb-prometheus-exporter > > ) > > > > 6. We could also just bundle this exporter (although it is written in Go, > > which we currently don’t have as a dependency. > > > > * * * > > > > Personally, I think the Prometheus format is widely enough used to warrant > > inclusion, as long as we do it tastefully. I think a new endpoint with an > > additional ?accept= or similar URL-level override for the format would be > > a pragmatic, if not entirely *neat* approach. If we can build this all in > > Erlang, the better, if we wanna shortcut dev time and bundle the Go > > project, I might be more hesitant. On the per-node-or-per-cluster > > question, I don’t know enough about the Prometheus format and whether it > > allows us to send the equivalent of {nodes: { “node1”: {…}, “node2”: {…}, > > “node3”: {…} }}, or whether it demands per-node output, in which case > > _active_tasks might get a bit awkward. > > > > Best > > Jan > > > > — > > Professional Support for Apache CouchDB: > > https://neighbourhood.ie/couchdb-support/ > > > > > > 24/7 Observation for your CouchDB Instances: > > https://opservatory.app > > > > > >> On 22. Sep 2020, at 14:55, jiangph <jiangpeng...@hotmail.com> wrote: > >> > >> Hey all, > >> > >> We would like to add a Prometheus metrics endpoint for CouchDB and > > wanted to see if the community would be interested in us contributing this > > to CouchDB 4.x. > >> > >> Prometheus is a CNCF open-source project and the Prometheus metrics > > endpoint format is supported by many monitoring tools. Its data model is > > based around having a metric name which then contains a label name and a > > label value: > >> > >> <metric name>{<label name>=<label value>, ...} > >> > >> And it supports the Counter, Gauge, Histogram, and Summary metric types. > > > >> > >> The idea for the new Prometheus endpoint, /_metrics, would be that the > > endpoint is a consolidation of the _stats [1], _system [2], and > > _active_tasks [3] endpoints. > >> > >> For _stats and _system, the conversion from JSON to Prometheus-based > > format seems to be straightforward. > >> > >> JSON format: > >> { > >> "value": { > >> "min": 0, > >> "max": 0, > >> "arithmetic_mean": 0, > >> "geometric_mean": 0, > >> "harmonic_mean": 0, > >> "median": 0, > >> "variance": 0, > >> "standard_deviation": 0, > >> ... > >> "percentile": [ > >> [ > >> 50, > >> 0 > >> ], > >> [ > >> 75, > >> 0 > >> ], > >> [ > >> 90, > >> 0 > >> ], > >> [ > >> 95, > >> 0 > >> ], > >> [ > >> 99, > >> 0 > >> ], > >> [ > >> 999, > >> 0 > >> ] > >> ], > >> "histogram": [ > >> [ > >> 0, > >> 0 > >> ] > >> ], > >> } > >> > >> Prometheus-based format: > >> > >> couchdb_stats{value="min"} 0 > >> couchdb_stats{value="max"} 0 > >> couchdb_stats{value="percentile50"} 0 > >> couchdb_stats{value="percentile75"} 0 > >> couchdb_stats{value="percentile95"} 0 > >> > >> For _active_tasks, the change will be a bit more complicated, and some > > fields will be added to labels and tags. > >> > >> JSON format: > >> > >> { > >> "checkpointed_source_seq": 68585, > >> "continuous": false, > >> "doc_id": null, > >> "doc_write_failures": 0, > >> "docs_read": 4524, > >> "docs_written": 4524, > >> "missing_revisions_found": 4524, > >> "pid": "<0.1538.5>", > >> "progress": 44, > >> "replication_id": "9bc1727d74d49d9e157e260bb8bbd1d5", > >> "revisions_checked": 4524, > >> "source": "mailbox", > >> "source_seq": 154419, > >> "started_on": 1376116644, > >> "target": " > > http://mailsrv:5984/mailbox > > < > > http://mailsrv:5984/mailbox > >> ", > >> "type": "replication", > >> "updated_on": 1376116651 > >> } > >> > >> Prometheus-based would look something like: > >> > >> format:couchdb_active_task{type="replication", source="mailbox", > > target=" > > http://mailsrv:5984/mailbox > > < > > http://mailsrv:5984/mailbox > >> ", docs_count = "docs_read"} 4524 > >> couchdb_active_task{type="replication", source="mailbox", target=" > > http://mailsrv:5984/mailbox > > < > > http://mailsrv:5984/mailbox > >> ", docs_count = "docs_written"} 4524 > >> couchdb_active_task{type="replication", source="mailbox", target=" > > http://mailsrv:5984/mailbox > > < > > http://mailsrv:5984/mailbox > >> ", docs_count = "missing_revisions_found"} 4524 > >> > >> > >> Best regards, > >> Garren Smith > >> Peng Hui Jiang > >> > >> [1] > > https://docs.couchdb.org/en/latest/api/server/common.html#node-node-name-stats > > > > < > > https://docs.couchdb.org/en/latest/api/server/common.html#node-node-name-stats > > > >> > >> [2] > > https://docs.couchdb.org/en/latest/api/server/common.html#active-tasks > > < > > https://docs.couchdb.org/en/latest/api/server/common.html#active-tasks > >> > >> [3] > > https://docs.couchdb.org/en/latest/api/server/common.html#node-node-name-system > > > > < > > https://docs.couchdb.org/en/latest/api/server/common.html#node-node-name-system > > > >> > > > > > > > > > > > > Unless stated otherwise above: > > IBM United Kingdom Limited - Registered in England and Wales with number > > 741598. > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > >
