[ https://issues.apache.org/jira/browse/KAFKA-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766592#comment-17766592 ]
Greg Harris commented on KAFKA-15473: ------------------------------------- Plugins could also appear multiple times <3.6.0-rc0 if multiple versions were on the plugin path concurrently. The DelegatingClassLoader would prefer the one with the latest version, but all of the different versions would be visible in the REST API. It also treated the undefined version as distinct from defined versions. Since 3.6.0 is also the first version with KAFKA-15291 and there are some public plugins which package the AK transforms, there are now going to be both un-versioned and versioned copies of the transforms classes. I think there are multiple ways to address this: 1. Show the source location of the plugins in the REST API to distinguish between apparently equivalent entries 2. Use the DelegatingClassLoader logic for choosing among multiple similarly-named plugins to only show the entry which the DelegatingClassLoader would use 3. Deduplicate the PluginInfos to avoid obvious repetition, but allow multiple versions to still be shown. (1) would require a KIP, and might be scope creep for the REST API. Theoretically the API client shouldn't care about the on-disk layout of plugins. (2) hides the duplicates introduced by KAFKA-15244 and KAFKA-15291, but also hides all but the latest version of each plugin. If someone has multiple versions of a plugin installed, they can currently diagnose that via REST API. After implementing solution (2), they would need to look at the worker logs. (3) hides the duplicates introduced by KAFKA-15244, but leaves the duplicates introduced by KAFKA-15291, and allows users to diagnose when multiple versions are installed. I'm not sure which of these to implement. In <3.6.0-rc0 it was not possible to diagnose installations with multiple copies of the same plugin when they had the same (or undefined) version, and in 3.6.0-rc0 that is now possible. If we implement solution (2) or (3), we would take away that capability. We might just be able to leave this as-is, and try to implement some form of solution (1) to make these duplicates more descriptive. > Connect connector-plugins endpoint shows duplicate plugins > ---------------------------------------------------------- > > Key: KAFKA-15473 > URL: https://issues.apache.org/jira/browse/KAFKA-15473 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect > Affects Versions: 3.6.0 > Reporter: Greg Harris > Assignee: Greg Harris > Priority: Major > Fix For: 3.6.0 > > > In <3.6.0-rc0, only one copy of each plugin would be shown. For example: > {noformat} > { > "class": "org.apache.kafka.connect.storage.StringConverter", > "type": "converter" > },{noformat} > In 3.6.0-rc0, there are multiple listings for the same plugin. For example: > > {noformat} > { > "class": "org.apache.kafka.connect.storage.StringConverter", > "type": "converter" > }, > { > "class": "org.apache.kafka.connect.storage.StringConverter", > "type": "converter" > }, > { > "class": "org.apache.kafka.connect.storage.StringConverter", > "type": "converter" > }, > { > "class": "org.apache.kafka.connect.storage.StringConverter", > "type": "converter" > }, > { > "class": "org.apache.kafka.connect.storage.StringConverter", > "type": "converter" > }, > { > "class": "org.apache.kafka.connect.storage.StringConverter", > "type": "converter", > "version": "3.6.0" > },{noformat} > These duplicates appear to happen when a plugin with the same class name > appears in multiple locations/classloaders. > When interpreting a connector configuration, only one of these plugins will > be chosen, so only one is relevant to show to users. The REST API should only > display the plugins which are eligible to be loaded, and hide the duplicates. -- This message was sent by Atlassian Jira (v8.20.10#820010)