gharris1727 commented on code in PR #14068: URL: https://github.com/apache/kafka/pull/14068#discussion_r1283591476
########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> + + <h5><a id="connect_plugindiscovery_migrateartifact" href="#connect_plugindiscovery_migrateartifact">Operators: Artifact Migration</a></h5> + + <p>As an operator of Connect, if you discover incompatible plugins, there are multiple ways to try to resolve the incompatibility. They are listed below from most to least preferable.</p> + + <ol> + <li>Upgrade your incompatible plugins to the latest release version from your plugin provider.</li> + <li>Contact your plugin provider and request that they migrate the plugin to be compatible, following the <a href="#connect_plugindiscovery_migratesource">source migration instructions</a>, and then upgrade to the migrated version.</li> + <li>Migrate the plugin artifacts yourself using the included migration script.</li> + </ol> + + <p>The migration script is located in <code>bin/connect-plugin-path.sh</code> and <code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation. The script can migrate incompatible plugin artifacts already installed on your Connect worker's <code>plugin.path</code> by adding or modifying JAR or resource files. This is not suitable for environments using code-signing, as this may change the artifacts such that they will fail signature verification. View the built-in help with <code>--help</code>.</p> + + <p>To perform a migration, first use the <code>list</code> subcommand to get an overview of the plugins available to the script. You must tell the script where to find plugins, which can be done with the repeatable <code>--worker-config</code>, <code>--plugin-path</code>, and <code>--plugin-location</code> arguments. The script will only migrate plugins present in the paths specified, so if you add plugins to your worker's classpath, then you will need to specify those plugins via one or more <code>--plugin-location</code> arguments.</p> Review Comment: > Maybe "Note that the script will ignore plugins present on the Connect worker's classpath unless they are contained in an explicitly-provided --plugin-location"? :+1: > Also, can you refresh my memory on why we've designed the script to ignore classpath plugins? It feels like this has made things more difficult, both for us (implementing the connect-plugin-path script) and users (adding a footgun for plugins installed directly on the classpath). The reason that the script doesn't migrate classpath plugins is because those containing JARs are part of the live-executing java process while the migration is running. I don't think editing them while they are being used is well defined, and would open us up to more bugs overall. And the script doesn't `list` them to have parity with `sync-manifests`. > IMO it's not too late to tweak this behavior if we decide it's better to take things in a different direction. I committed to it in the KIP: > This script will only list and migrate plugins which are on the plugin.path of a Connect worker, and are loaded in isolation. This script will not list or migrate plugins which are included on the classpath, and will assume that classpath plugins have manifests added through some other method. I understand that we can change the KIP, but I don't think we should. I think we might also need to change the name of the script too :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org