Shinjo Wang created NIFI-16481:
----------------------------------
Summary: Kafka3ConnectionService cannot be enabled until restart
after a dynamic property is set through the REST API: NoClassDefFoundError for
ProducerConfig
Key: NIFI-16481
URL: https://issues.apache.org/jira/browse/NIFI-16481
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework, Extensions
Affects Versions: 2.12.0, 2.11.0, 2.6.0
Environment: NiFi 2.6.0, 2.11.0, 2.12.0 (apache/nifi Docker images,
linux/arm64), standalone, single-user authentication
Not reproducible on NiFi 2.5.0 (kafka-clients 4.0.0)
Reporter: Shinjo Wang
h3. Summary
After NiFi starts, if the first code path that initializes {{ProducerConfig}}
in {{nifi-kafka-3-service-nar}} is a REST request that creates or updates a
Kafka3ConnectionService with a dynamic property (for example
{{metadata.max.age.ms}}), the request returns HTTP 500. After that, no
Kafka3ConnectionService can be enabled until NiFi is restarted. Services stay
in ENABLING with {{NoClassDefFoundError: Could not initialize class
org.apache.kafka.clients.producer.ProducerConfig}}.
This affects NiFi 2.6.0 and later (kafka-clients 4.1.0 and later).
h3. Steps to Reproduce
# Start NiFi with an empty flow, for example {{apache/nifi:2.12.0}}. Do not
add, enable, or verify any Kafka component first.
# Create a Kafka3ConnectionService with a dynamic property. This returns HTTP
500.
{code:bash}
curl -sk -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json'
-X POST \
https://localhost:8443/nifi-api/process-groups/$PG/controller-services -d '{
"revision":{"version":0},
"component":{
"type":"org.apache.nifi.kafka.service.Kafka3ConnectionService",
"properties":{"bootstrap.servers":"localhost:9092","metadata.max.age.ms":"60000"}}}'
{code}
# Create a Kafka3ConnectionService without dynamic properties and enable it. It
stays in ENABLING.
If a Kafka3ConnectionService without dynamic properties is enabled before step
2, the problem does not occur. The UI does not reproduce it, probably because
it calls {{GET /controller-services/\{id\}/descriptors}} first, which uses the
NAR class loader.
h3. Errors
{noformat}
ERROR [NiFi Web Server-31] o.a.nifi.web.api.config.ThrowableMapper An
unexpected error has occurred: java.lang.ExceptionInInitializerError. Returning
Internal Server Error response.
java.lang.ExceptionInInitializerError: null
at
org.apache.nifi.kafka.shared.property.provider.StandardKafkaPropertyNameProvider.<init>(StandardKafkaPropertyNameProvider.java:48)
at
org.apache.nifi.kafka.shared.validation.DynamicPropertyValidator.<init>(DynamicPropertyValidator.java:39)
at
org.apache.nifi.kafka.service.Kafka3ConnectionService.getSupportedDynamicPropertyDescriptor(Kafka3ConnectionService.java:219)
at
org.apache.nifi.components.AbstractConfigurableComponent.getPropertyDescriptor(AbstractConfigurableComponent.java:56)
at
org.apache.nifi.authorization.StandardAuthorizableLookup$ConfigurableComponentAuthorizable.getPropertyDescriptor(StandardAuthorizableLookup.java:871)
at
org.apache.nifi.authorization.AuthorizeControllerServiceReference.authorizeControllerServiceReferences(AuthorizeControllerServiceReference.java:91)
at
org.apache.nifi.web.api.ProcessGroupResource.lambda$createControllerService$55(ProcessGroupResource.java:2557)
...
Caused by: org.apache.kafka.common.config.ConfigException: Invalid value
org.apache.kafka.common.security.oauthbearer.DefaultJwtRetriever for
configuration sasl.oauthbearer.jwt.retriever.class: Class
org.apache.kafka.common.security.oauthbearer.DefaultJwtRetriever could not be
found.
at
org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:778)
...
at
org.apache.kafka.clients.producer.ProducerConfig.<clinit>(ProducerConfig.java:518)
{noformat}
h3. Root Cause
# *The framework calls component code without the component NAR class loader
during authorization.* {{AuthorizeControllerServiceReference}} calls
{{getPropertyDescriptor(name)}} for each requested property.
{{StandardAuthorizableLookup.ConfigurableComponentAuthorizable}} (create) and
{{ControllerServiceComponentAuthorizable}} (update) call the component
directly, so the call runs with the Jetty thread context class loader.
{{AbstractComponentNode.getPropertyDescriptor}} and
{{ProcessorComponentAuthorizable}} already switch to the NAR class loader. The
authorizables for Reporting Tasks, Flow Analysis Rules, Parameter Providers,
and Flow Registry Clients have the same gap.
# *ProducerConfig static initialization depends on the thread context class
loader in kafka-clients 4.1.0 and later.* For an undefined property name,
Kafka3ConnectionService creates
{{DynamicPropertyValidator(ProducerConfig.class, ...)}}, which reads the static
fields of {{ProducerConfig}}. Since kafka-clients 4.1.0, the static initializer
resolves the default {{DefaultJwtRetriever}} class through the thread context
class loader (KAFKA-21107). The Jetty class loader cannot see kafka-clients, so
initialization fails, and the JVM keeps {{ProducerConfig}} unusable in that NAR
class loader.
NiFi 2.5.0 has the same framework code path, but kafka-clients 4.0.0 has no
such default, so it does not fail. The upgrade to kafka-clients 4.1.0 in
NIFI-14937 exposed the problem. The dynamic property code path was added in
NIFI-14271.
h3. Workaround
After a restart, enable a Kafka3ConnectionService without dynamic properties
before setting a dynamic property through the REST API.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
