Stephen Darlington created IGNITE-13996:
-------------------------------------------
Summary: JMX beans in a format not understood by Prometheus
Key: IGNITE-13996
URL: https://issues.apache.org/jira/browse/IGNITE-13996
Project: Ignite
Issue Type: Bug
Affects Versions: 2.9.1, 2.8.1, 2.9
Reporter: Stephen Darlington
When I connect Ignite to Prometheus' JMX exporter, I get the following
exception:
{{ }}
{{ Jan 13, 2021 5:31:49 PM
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector collect}}
{{ SEVERE: JMX scrape failed: java.lang.IllegalArgumentException: Not an
Attribute:
javax.management.openmbean.TabularDataSupport(tabularType=javax.management.openmbean.TabularType(name=org.apache.ignite.spi.systemview.view.ScanQueryView,rowType=javax.management.openmbean.CompositeType(name=org.apache.ignite.spi.systemview.view.ScanQueryView,items=((itemName=cacheGroupId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=cacheGroupName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=cacheId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=cacheName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=canceled,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=duration,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=filter,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=keepBinary,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=local,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=originNodeId,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=pageSize,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=partition,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=queryId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=startTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=subjectId,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=systemViewRowId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=taskName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=topology,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=transformer,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)))),indexNames=(systemViewRowId)),contents={})}}
{{ at javax.management.AttributeList.adding(AttributeList.java:328)}}
{{ at javax.management.AttributeList.adding(AttributeList.java:335)}}
{{ at javax.management.AttributeList.asList(AttributeList.java:165)}}
{{ at
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:156)}}
{{ at
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:117)}}
{{ at
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:460)}}
{{ at
io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)}}
{{ at
io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216)}}
{{ at
io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137)}}
{{ at
io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)}}
{{ at
io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:59)}}
{{ at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)}}
{{ at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)}}
{{ at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)}}
{{ at
sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)}}
{{ at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)}}
{{ at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)}}
{{ at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
{{ at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
{{ at java.lang.Thread.run(Thread.java:748)}}
{{ }}
And only the basic JVM metrics appear in Prometheus. A workaround would be to
use OpenCensus, but a lot of people seem to prefer JMX.
It's not clear to me Ignite's JMX output is incorrect ([Prometheus' developers
don't seem keen to resolve on their
side|https://github.com/prometheus/jmx_exporter/issues/483]) but ideally,
Ignite would work correctly with a common monitoring tool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
