Hello everyone and happy holidays, The changes below are ready for review! Benchmarks are also inside.
Expose all table metrics in virtual tables https://issues.apache.org/jira/browse/CASSANDRA-14572 https://github.com/apache/cassandra/pull/2958/files On Tue, 12 Dec 2023 at 22:05, Maxim Muzafarov <mmu...@apache.org> wrote: > > Hello everyone, > > > I still think Cassandra will benefit from having this idea implemented > and used through the source code, so I've done another round of > rethinking this concept and it seems I've found a solution. As a > result, we can significantly reduce the cost of implementing and > maintaining both new and existing virtual tables and make our users > happier by seeing everything they need through virtual tables. > > So, I think we should limit the scope of the original proposal to the > following: > ## A framework for exposing any internal data collection to virtual > tables ONLY. ## > > As a proof of concept, I took the CASSANDRA-14572 "Expose all table > metrics in virtual table" JIRA ticket, which provides a good > opportunity to demonstrate how to export all metrics to VTs at once > without having boilerplate implementations. Currently, we already have > CQLMetricsTable, BatchMetricsTable, etc. that expose metrics to VTs in > a pretty similar way, and most of the metrics groups are located under > the org.apache.cassandra.metrics package still lacks their > representation as VTs either. I've used the MetricRegistry collection > as a view of registered metrics to export them to VT using the > prototype accordingly. > > The prototype is complete. You can run a node locally and check the > available virtual tables with cqlsh, or you can check the changes > using the following link to the PR: > https://github.com/apache/cassandra/pull/2958/files > > > Below are some key points about the design itself: > > 1. All new virtual tables with metrics have "metric" as a prefix so > that they are fairly easy to find using TAB on the cqlsh command line. > Metrics are split into virtual tables as they are listed in the > org.apache.cassandra.metrics e.g. metrics_cql, metrics_tcm etc. In > addition, they are also grouped by metric type e.g. > metric_type_histogram, metric_type_counter etc. There is a table > called "metric_all_metric_groups" with all available metric groups. > > 2. To create a new virtual table representation of an internal > collection a developer needs to do two things: create a virtual table > row representation, and register it using > CollectionVirtualTableAdapter, which acts as an adapter between > internal data and a virtual table. Here's how I did it for the thread > pools VT, this is a fully backward compatible change: > https://github.com/apache/cassandra/pull/2958/files#diff-5fda13a633723cdf232bba465e6fb7ab74cdc02f7ba55dae4d1cf494ffb71abaR61 > > 3. The "metrics_keyspace" virtual table ended up being quite large > since it contains all the metrics for all available keyspaces on a > local node, so the default implementation provided by > AbstractVirtualTable is not suitable for the proposal. The > AbstractVirtualTable materializes a full data collection on the heap > using SimpleDataSet, regardless of the portion of data that is being > queried. In this case, we have to use an iterative approach, as the > CollectionVirtualTableAdapter does (the problem was discussed in > CASSANDRA-14629 and is now a part of the solution). This also helps to > keep the memory footprint low. > > 4. Another valuable change is the CassandraMetricsRegistry itself. The > problem here is that the metrics and their aliases are currently > exported to JMX, but the implemented virtual tables export the metrics > in their way and most of the cases don't respect the metric aliases > which are registered in the MetricsRegistry. This should be fixed as a > part of the CASSANDRA-14572 to avoid ambiguity for all known metrics > once and for all. > > Here are the links to the issue and the PR: > https://issues.apache.org/jira/browse/CASSANDRA-14572 > https://github.com/apache/cassandra/pull/2958/files > > > I'm excited about how these changes look right now, so please share > your feedback and thoughts. > The PR lacks good test coverage, I'll fix it as soon as we have a > clear vision of the design (or much sooner) :-) > > On Mon, 30 Jan 2023 at 17:43, David Capwell <dcapw...@apple.com> wrote: > > > > I *think* this is arguably true for a vtable / CQL-based solution as well > > from the "you don't know how people are using your API" perspective. > > > > > > Very fair point and think that justifies a different thread to talk about > > backwards compatibility for our tables (virtual and not); maybe we can lump > > together the JMX backwards compatibility conversation as well in that new > > thread. > > > > On Jan 28, 2023, at 4:09 PM, Josh McKenzie <jmcken...@apache.org> wrote: > > > > First off - thanks so much for putting in this effort Maxim! This is > > excellent work. > > > > Some thoughts on the CEP and responses in thread: > > > > Considering that JMX is usually not used and disabled in production > > environments for various performance and security reasons, the operator may > > not see the same picture from various of Dropwizard's metrics exporters and > > integrations as Cassandra's JMX metrics provide [1][2]. > > > > I don't think this assertion is true. Cassandra is running in a lot of > > places in the world, and JMX has been in this ecosystem for a long time; we > > need data that is basically impossible to get to claim "JMX is usually not > > used in C* environments in prod". > > > > I also wonder about if we should care about JMX? I know many wish to > > migrate (its going to be a very long time) away from JMX, so do we need a > > wrapper to make JMX and vtables consistent? > > > > If we can move away from a bespoke vtable or JMX based implementation and > > instead have a templatized solution each of these is generated from, that > > to me is the superior option. There's little harm in adding new JMX > > endpoints (or hell, other metrics framework integration?) as a byproduct of > > adding new vtable exposed metrics; we have the same maintenance obligation > > to them as we have to the vtables and if it generates from the same base > > data, we shouldn't have any further maintenance burden due to its presence > > right? > > > > we wish to move away from JMX > > > > I do, and you do, and many people do, but I don't believe all people on the > > project do. The last time this came up in slack the conclusion was "Josh > > should go draft a CEP to chart out a path to moving off JMX while > > maintaining backwards-compat w/existing JMX metrics for environments that > > are using them" (so I'm excited to see this CEP pop up before I got to it! > > ;)). Moving to a system that gives us a 0-cost way to keep JMX and vtable > > in sync over time on new metrics seems like a nice compromise for folks > > that have built out JMX-based maintenance infra on top of C*. Plus removing > > the boilerplate toil on vtables. win-win. > > > > If we add a column to the end of the JMX row did we just break users? > > > > I *think* this is arguably true for a vtable / CQL-based solution as well > > from the "you don't know how people are using your API" perspective. Unless > > we have clear guidelines about discretely selecting the columns you want > > from a vtable and trust users to follow them, if people have brittle greedy > > parsers pulling in all data from vtables we could very well break them as > > well by adding a new column right? Could be wrong here; I haven't written > > anything that consumes vtable metric data and maybe the obvious idiom in > > the face of that is robust in the presence of column addition. /shrug > > > > It's certainly more flexible and simpler to write to w/out detonating > > compared to JMX, but it's still an API we'd be revving. > > > > On Sat, Jan 28, 2023, at 4:24 PM, Ekaterina Dimitrova wrote: > > > > Overall I have similar thoughts and questions as David. > > > > I just wanted to add a reminder about this thread from last summer[1]. We > > already have issues with the alignment of JMX and Settings Virtual Table. I > > guess this is how Maxim got inspired to suggest this framework proposal > > which I want to thank him for! (I noticed he assigned CASSANDRA-15254) > > > > Not to open the Pandora box, but to me the most important thing here is to > > come into agreement about the future of JMX and what we will do or not as a > > community. Also, how much time people are able to invest. I guess this will > > influence any directions to be taken here. > > > > [1] > > https://lists.apache.org/thread/8mjcwdyqoobpvw2262bqmskkhs76pp69 > > > > > > On Thu, 26 Jan 2023 at 12:41, David Capwell <dcapw...@apple.com> wrote: > > > > I took a look and I see the result is an interface that looks like the > > vtable interface, that is then used by vtables and JMX? My first thought > > is why not just use the vtable logic? > > > > I also wonder about if we should care about JMX? I know many wish to > > migrate (its going to be a very long time) away from JMX, so do we need a > > wrapper to make JMX and vtables consistent? I am cool with something like > > the following > > > > registerWithJMX(jmxName, query(“SELECT * FROM system_views.streaming”)); > > > > > > So if we want to have a JMX view that matches the table then that’s cool by > > me, but one thing that has been brought up in reviews is backwards > > compatibility with regard to adding columns… If we add a column to the end > > of the JMX row did we just break users? > > > > Considering that JMX is usually not used and disabled in production > > environments for various performance and security reasons, the operator may > > not see the same picture from various of Dropwizard's metrics exporters > > > > If this is a real problem people are hitting, we can always add the ability > > to push metrics to common systems with a pluggable way to add non-standard > > solutions. Dropwizard already support this so would be low hanging fruit > > to address this. > > > > To make the proposed changes backwards compatible with the previous version > > of Cassandra, all MBeans and Virtual Tables we already have will remain > > unchanged > > > > > > If this is for new JMX endpoints moving forward, I am not sure of the > > benefit for the same reason listed above; we wish to move away from JMX > > > > On Jan 25, 2023, at 10:51 AM, Maxim Muzafarov <mmu...@apache.org> wrote: > > > > Hello Cassandra Community, > > > > > > I've been faced with a number of inconsistencies in the user APIs of > > the internal data collections representation exposed through the > > Cassandra monitoring interfaces that need to be fully aligned from an > > operator perspective. First of all, I'm highlighting JMX, Dropwizard > > Metrics, and Virtual Tables user interfaces. In order to address all > > these inconsistencies, I have created a draft enhancement proposal > > that describes everything I have found and how we can fix it once and > > for all. > > > > I'd like to hear your opinion and thoughts on it. Please take a look: > > https://docs.google.com/document/d/1j4J3bPWjQkAU9x4G-zxKObxPrKg36jLRT6xpUoNJa8Q > > > > > > -- > > Maxim Muzafarov > > > >
