gharris1727 commented on code in PR #14068: URL: https://github.com/apache/kafka/pull/14068#discussion_r1283562840
########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> Review Comment: I think an unpublished draft had the connect-plugin-path tool as a third verification option. I removed it because even though it is easier, it has the risk of giving false negatives in the presence of operator mistakes. I want these verification steps to be as reliable as possible, since a false negative could mean an outage in production. The script does not check classpath plugins, and the classpath for the script execution may be different than for the connect worker. If someone has a non-compatible plugin added to the CLASSPATH in their production environment, and does not explicitly include that as a `--plugin-location` in the script, then the script will not check that plugin, and will not detect those incompatible plugins. Later in the docs I call out this special case for the CLASSPATH, relying on the more reliable verification steps as a backstop. If we recommend using `list` for verification, the user can make the same mistake in both places and not catch their mistake. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org