exceptionfactory commented on code in PR #7124: URL: https://github.com/apache/nifi/pull/7124#discussion_r1178132789
########## nifi-docs/src/main/asciidoc/administration-guide.adoc: ########## @@ -4824,3 +4824,32 @@ nifi.diagnostics.on.shutdown.max.directory.size=10 MB ``` In the case of a lengthy diagnostic, NiFi may terminate before the command execution ends. In this case, the `graceful.shutdown.seconds` property should be set to a higher value in the `bootstrap.conf` configuration file. + +[[jmx_bean_metrics]] +== JMX Bean metrics + +It is possible to get JMX metrics data calling `<your_nifi_url>/nifi-api/jmx-metrics` with 'Flow' 'Read' and 'SystemDiagnostics' 'Read' permissions. + +The information available depends on the registered MBeans and usually related to performance indicators such as request/response rate, latency, request size, request counts or tool specific ones like commit rate or sync/join time for Kafka. + +Listing of certain MBeans can be prohibited by a regex expression and can be configured by a NiFi application property. Leaving the property empty means all the MBeans may appear in the result list. Default value is to display all MBeans. Review Comment: ```suggestion Listing of certain MBeans can be prohibited using a regular expression pattern in application properties. Leaving the property empty means all the MBeans may be returned. The default value returns all MBeans. ``` ########## nifi-docs/src/main/asciidoc/administration-guide.adoc: ########## @@ -4824,3 +4824,32 @@ nifi.diagnostics.on.shutdown.max.directory.size=10 MB ``` In the case of a lengthy diagnostic, NiFi may terminate before the command execution ends. In this case, the `graceful.shutdown.seconds` property should be set to a higher value in the `bootstrap.conf` configuration file. + +[[jmx_bean_metrics]] +== JMX Bean metrics + +It is possible to get JMX metrics data calling `<your_nifi_url>/nifi-api/jmx-metrics` with 'Flow' 'Read' and 'SystemDiagnostics' 'Read' permissions. + +The information available depends on the registered MBeans and usually related to performance indicators such as request/response rate, latency, request size, request counts or tool specific ones like commit rate or sync/join time for Kafka. Review Comment: ```suggestion The information available depends on the registered MBeans. Metrics can contain data related to performance indicators. ``` ########## nifi-docs/src/main/asciidoc/administration-guide.adoc: ########## @@ -4824,3 +4824,32 @@ nifi.diagnostics.on.shutdown.max.directory.size=10 MB ``` In the case of a lengthy diagnostic, NiFi may terminate before the command execution ends. In this case, the `graceful.shutdown.seconds` property should be set to a higher value in the `bootstrap.conf` configuration file. + +[[jmx_bean_metrics]] +== JMX Bean metrics + +It is possible to get JMX metrics data calling `<your_nifi_url>/nifi-api/jmx-metrics` with 'Flow' 'Read' and 'SystemDiagnostics' 'Read' permissions. Review Comment: Recommend adjusting the wording: ```suggestion It is possible to get JMX metrics using the REST API using the `/nifi-api/jmx-metrics` path with read permissions on flow and system diagnostics resources. ``` ########## nifi-docs/src/main/asciidoc/administration-guide.adoc: ########## @@ -4824,3 +4824,32 @@ nifi.diagnostics.on.shutdown.max.directory.size=10 MB ``` In the case of a lengthy diagnostic, NiFi may terminate before the command execution ends. In this case, the `graceful.shutdown.seconds` property should be set to a higher value in the `bootstrap.conf` configuration file. + +[[jmx_bean_metrics]] +== JMX Bean metrics + +It is possible to get JMX metrics data calling `<your_nifi_url>/nifi-api/jmx-metrics` with 'Flow' 'Read' and 'SystemDiagnostics' 'Read' permissions. + +The information available depends on the registered MBeans and usually related to performance indicators such as request/response rate, latency, request size, request counts or tool specific ones like commit rate or sync/join time for Kafka. + +Listing of certain MBeans can be prohibited by a regex expression and can be configured by a NiFi application property. Leaving the property empty means all the MBeans may appear in the result list. Default value is to display all MBeans. + + nifi.jmx.metrics.blocked.filter.pattern=prohibited.bean1|prohibited.bean2 + + +An optionally provided query parameter using a regex expression, will display only MBeans with matching names. Leaving this parameter empty means listing all MBeans except the ones filtered out by the security filter. + + https://nifihost.com:8443/nifi-api/jmx-metrics?beanNameFilter=bean.name.1|bean.name.2 Review Comment: ```suggestion https://localhost:8443/nifi-api/jmx-metrics?beanNameFilter=bean.name.1|bean.name.2 ``` ########## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/metrics/jmx/JmxMetricsFilter.java: ########## @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.metrics.jmx; + +import org.apache.nifi.web.api.dto.JmxMetricsResultDTO; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Collection; +import java.util.regex.Pattern; +import java.util.regex.PatternSyntaxException; +import java.util.stream.Collectors; + +public class JmxMetricsFilter { + private static final Logger LOGGER = LoggerFactory.getLogger(JmxMetricsFilter.class); + private final static String MATCH_NOTHING = "~^"; + private final static String MATCH_ALL = ""; + private final Pattern blockedNameFilter; + private final Pattern beanNameFilter; + + public JmxMetricsFilter(final String blockedNameFilter, final String beanNameFilter) { + this.blockedNameFilter = createPattern(blockedNameFilter, MATCH_NOTHING); + this.beanNameFilter = createPattern(beanNameFilter, MATCH_ALL); + } + + private Pattern createPattern(final String filter, final String defaultValue) { + try { + if (filter == null || filter.isEmpty()) { + return Pattern.compile(defaultValue); + } else { + return Pattern.compile(filter); + } + } catch (PatternSyntaxException e) { + LOGGER.warn("Invalid filter {} , will use default filtering.", filter); Review Comment: Recommend adjusting the wording. ```suggestion LOGGER.warn("Invalid JMX MBean filter pattern ignored [{}]", filter); ``` ########## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/metrics/jmx/JmxMetricsService.java: ########## @@ -0,0 +1,25 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.metrics.jmx; + +import org.apache.nifi.web.api.dto.JmxMetricsResultDTO; + +import java.util.Collection; + +public interface JmxMetricsService { + Collection<JmxMetricsResultDTO> getFilteredMBeanMetrics(final String beanNameFilter); Review Comment: It would be helpful to add a JavaDoc comment to this method, indicating that the `beanNameFilter` is expected to contain a Regular Expression Pattern. ########## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/metrics/jmx/StandardJmxMetricsService.java: ########## @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.metrics.jmx; + +import org.apache.nifi.util.NiFiProperties; +import org.apache.nifi.web.api.dto.JmxMetricsResultDTO; + +import java.util.Collection; + +public class StandardJmxMetricsService implements JmxMetricsService { + private static final String JMX_METRICS_NIFI_PROPERTY = "nifi.jmx.metrics.blocked.filter.pattern"; Review Comment: This property name should be moved to a public static value in the `NiFiProperties` class. ########## nifi-docs/src/main/asciidoc/administration-guide.adoc: ########## @@ -4824,3 +4824,32 @@ nifi.diagnostics.on.shutdown.max.directory.size=10 MB ``` In the case of a lengthy diagnostic, NiFi may terminate before the command execution ends. In this case, the `graceful.shutdown.seconds` property should be set to a higher value in the `bootstrap.conf` configuration file. + +[[jmx_bean_metrics]] +== JMX Bean metrics + +It is possible to get JMX metrics data calling `<your_nifi_url>/nifi-api/jmx-metrics` with 'Flow' 'Read' and 'SystemDiagnostics' 'Read' permissions. + +The information available depends on the registered MBeans and usually related to performance indicators such as request/response rate, latency, request size, request counts or tool specific ones like commit rate or sync/join time for Kafka. + +Listing of certain MBeans can be prohibited by a regex expression and can be configured by a NiFi application property. Leaving the property empty means all the MBeans may appear in the result list. Default value is to display all MBeans. + + nifi.jmx.metrics.blocked.filter.pattern=prohibited.bean1|prohibited.bean2 + + +An optionally provided query parameter using a regex expression, will display only MBeans with matching names. Leaving this parameter empty means listing all MBeans except the ones filtered out by the security filter. Review Comment: ```suggestion An optionally provided query parameter using a regular expression pattern, will display only MBeans with matching names. Leaving this parameter empty means listing all MBeans except those filtered out by the blocked filter pattern. ``` ########## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/metrics/jmx/JmxMetricsCollector.java: ########## @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api.metrics.jmx; + +import org.apache.nifi.web.api.dto.JmxMetricsResultDTO; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.management.AttributeNotFoundException; +import javax.management.InstanceNotFoundException; +import javax.management.IntrospectionException; +import javax.management.MBeanAttributeInfo; +import javax.management.MBeanException; +import javax.management.MBeanInfo; +import javax.management.MBeanServer; +import javax.management.MalformedObjectNameException; +import javax.management.ObjectInstance; +import javax.management.ObjectName; +import javax.management.ReflectionException; +import javax.management.RuntimeMBeanException; +import java.lang.management.ManagementFactory; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Set; + +public class JmxMetricsCollector { + private static final Logger LOGGER = LoggerFactory.getLogger(JmxMetricsCollector.class); + private final static String PATTERN_FOR_ALL_OBJECT_NAMES = "*:*"; + private final JmxMetricsResultConverter resultConverter; + + public JmxMetricsCollector(final JmxMetricsResultConverter metricsResultConverter) { + this.resultConverter = metricsResultConverter; + } + + public Collection<JmxMetricsResultDTO> getBeanMetrics() { + final MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer(); + final Set<ObjectInstance> instances; + try { + instances = mBeanServer.queryMBeans(new ObjectName(PATTERN_FOR_ALL_OBJECT_NAMES), null); + } catch (MalformedObjectNameException e) { + throw new RuntimeException("Invalid ObjectName pattern", e); + } + + final Collection<JmxMetricsResultDTO> results = new ArrayList<>(); + for (final ObjectInstance instance : instances) { + final MBeanInfo info; + try { + info = mBeanServer.getMBeanInfo(instance.getObjectName()); + } catch (InstanceNotFoundException | ReflectionException | IntrospectionException e) { + continue; + } + + for (MBeanAttributeInfo attribute : info.getAttributes()) { + try { + final String beanName = instance.getObjectName().getCanonicalName(); + final String attributeName = attribute.getName(); + final Object attributeValue = resultConverter.convert(mBeanServer.getAttribute(instance.getObjectName(), attribute.getName())); + + results.add(new JmxMetricsResultDTO(beanName, attributeName, attributeValue)); + } catch (MBeanException | RuntimeMBeanException | ReflectionException | InstanceNotFoundException | AttributeNotFoundException e) { + //Empty or invalid attributes should not stop the loop. + LOGGER.trace("Invalid attribute {}", attribute.getName()); Review Comment: It would be helpful to add a little more context to the message: ```suggestion LOGGER.debug("MBean Object [{}] invalid attribute [{}] found", instance.getObjectName(), attribute.getName()); ``` ########## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/api/TestJmxMetricsResource.java: ########## @@ -0,0 +1,348 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.web.api; + +import org.apache.nifi.util.NiFiProperties; +import org.apache.nifi.web.NiFiServiceFacade; +import org.apache.nifi.web.api.dto.JmxMetricsResultDTO; +import org.apache.nifi.web.api.entity.JmxMetricsResultsEntity; +import org.apache.nifi.web.api.metrics.jmx.JmxMetricsCollector; +import org.apache.nifi.web.api.metrics.jmx.JmxMetricsResultConverter; +import org.apache.nifi.web.api.metrics.jmx.StandardJmxMetricsService; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import javax.management.InstanceAlreadyExistsException; +import javax.management.InstanceNotFoundException; +import javax.management.MBeanRegistrationException; +import javax.management.MBeanServer; +import javax.management.MalformedObjectNameException; +import javax.management.NotCompliantMBeanException; +import javax.management.ObjectName; +import javax.management.openmbean.CompositeData; +import javax.management.openmbean.CompositeDataSupport; +import javax.management.openmbean.CompositeType; +import javax.management.openmbean.OpenDataException; +import javax.management.openmbean.OpenType; +import javax.management.openmbean.SimpleType; +import javax.management.openmbean.TabularData; +import javax.management.openmbean.TabularDataSupport; +import javax.management.openmbean.TabularType; +import java.lang.management.ManagementFactory; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Map; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +@ExtendWith(MockitoExtension.class) +public class TestJmxMetricsResource { + private static final String JMX_METRICS_NIFI_PROPERTY = "nifi.jmx.metrics.blocked.filter.pattern"; Review Comment: After moving the property name to `NiFiProperties`, this can be removed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
