C0urante commented on code in PR #14068: URL: https://github.com/apache/kafka/pull/14068#discussion_r1283295501
########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> Review Comment: I don't think we should explicitly call out the `EmbeddedConnectCluster` class, since it's not public API and discussing it in our docs may mistakenly give people the impression that it is. Perhaps we could mention the general utility of the `HYBRID_FAIL` mode in testing environments without providing specific details of how we use it ourselves? ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> Review Comment: Does "Connectors and Tasks" cover anything that just "connectors" doesn't? ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> Review Comment: Can be more explicit: "compatible with `SERVICE_LOAD` mode" instead of just "compatible. ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> + + <h5><a id="connect_plugindiscovery_migrateartifact" href="#connect_plugindiscovery_migrateartifact">Operators: Artifact Migration</a></h5> + + <p>As an operator of Connect, if you discover incompatible plugins, there are multiple ways to try to resolve the incompatibility. They are listed below from most to least preferable.</p> + + <ol> + <li>Upgrade your incompatible plugins to the latest release version from your plugin provider.</li> + <li>Contact your plugin provider and request that they migrate the plugin to be compatible, following the <a href="#connect_plugindiscovery_migratesource">source migration instructions</a>, and then upgrade to the migrated version.</li> + <li>Migrate the plugin artifacts yourself using the included migration script.</li> + </ol> + + <p>The migration script is located in <code>bin/connect-plugin-path.sh</code> and <code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation. The script can migrate incompatible plugin artifacts already installed on your Connect worker's <code>plugin.path</code> by adding or modifying JAR or resource files. This is not suitable for environments using code-signing, as this may change the artifacts such that they will fail signature verification. View the built-in help with <code>--help</code>.</p> + + <p>To perform a migration, first use the <code>list</code> subcommand to get an overview of the plugins available to the script. You must tell the script where to find plugins, which can be done with the repeatable <code>--worker-config</code>, <code>--plugin-path</code>, and <code>--plugin-location</code> arguments. The script will only migrate plugins present in the paths specified, so if you add plugins to your worker's classpath, then you will need to specify those plugins via one or more <code>--plugin-location</code> arguments.</p> Review Comment: This part seems a little unclear: "The script will only migrate plugins present in the paths specified, so if you add plugins to your worker's classpath, then you will need to specify those plugins via one or more <code>--plugin-location</code> arguments." Maybe "Note that the script will ignore plugins present on the Connect worker's classpath unless they are contained in an explicitly-provided <code>--plugin-location</code>"? Also, can you refresh my memory on why we've designed the script to ignore classpath plugins? It feels like this has made things more difficult, both for us (implementing the `connect-plugin-path` script) and users (adding a footgun for plugins installed directly on the classpath). IMO it's not too late to tweak this behavior if we decide it's better to take things in a different direction. ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> Review Comment: Nit: "set of installed plugins is compatible" instead of "set of installed plugins are compatible" ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> + + <h5><a id="connect_plugindiscovery_migrateartifact" href="#connect_plugindiscovery_migrateartifact">Operators: Artifact Migration</a></h5> + + <p>As an operator of Connect, if you discover incompatible plugins, there are multiple ways to try to resolve the incompatibility. They are listed below from most to least preferable.</p> + + <ol> + <li>Upgrade your incompatible plugins to the latest release version from your plugin provider.</li> Review Comment: This is a little misleading; someone may stop reading here and assume that an upgrade is guaranteed to fix the problem. Maybe "Check the latest release of your plugin and, if it is compatible, upgrade to that version"? ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> + + <h5><a id="connect_plugindiscovery_migrateartifact" href="#connect_plugindiscovery_migrateartifact">Operators: Artifact Migration</a></h5> + + <p>As an operator of Connect, if you discover incompatible plugins, there are multiple ways to try to resolve the incompatibility. They are listed below from most to least preferable.</p> + + <ol> + <li>Upgrade your incompatible plugins to the latest release version from your plugin provider.</li> + <li>Contact your plugin provider and request that they migrate the plugin to be compatible, following the <a href="#connect_plugindiscovery_migratesource">source migration instructions</a>, and then upgrade to the migrated version.</li> + <li>Migrate the plugin artifacts yourself using the included migration script.</li> + </ol> + + <p>The migration script is located in <code>bin/connect-plugin-path.sh</code> and <code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation. The script can migrate incompatible plugin artifacts already installed on your Connect worker's <code>plugin.path</code> by adding or modifying JAR or resource files. This is not suitable for environments using code-signing, as this may change the artifacts such that they will fail signature verification. View the built-in help with <code>--help</code>.</p> + + <p>To perform a migration, first use the <code>list</code> subcommand to get an overview of the plugins available to the script. You must tell the script where to find plugins, which can be done with the repeatable <code>--worker-config</code>, <code>--plugin-path</code>, and <code>--plugin-location</code> arguments. The script will only migrate plugins present in the paths specified, so if you add plugins to your worker's classpath, then you will need to specify those plugins via one or more <code>--plugin-location</code> arguments.</p> + + <p>Once you see that all incompatible plugins are included in the listing, you can proceed to dry-run the migration with <code>sync-manifests --dry-run</code>. This will perform all parts of the migration, except for writing the results of the migration to disk. Note that the <code>sync-manifests</code> command requires all specified paths to be writable, and may alter the contents of the directories. Make a backup of the specified paths, or copy them to a writable directory.</p> + + <p>Ensure that you have a backup and the dry-run succeeds before removing the <code>--dry-run</code> flag and actually running the migration. If the migration fails without the <code>--dry-run</code> flag, then the partially migrated artifacts should be discarded. The migration is idempotent, so running it multiple times and on already-migrated plugins is safe. After the migration is completed, you should <a href="#connect_plugindiscovery_compatibility">verify the migration is complete</a>. These migration and verification steps may be put into a Continuous Integration process to automatically migrate plugins.</p> Review Comment: In "Ensure that you have a backup", it's a little unclear what "backup" refers to. Perhaps "Ensure that you have a backup of your plugins"? ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> + + <h5><a id="connect_plugindiscovery_migrateartifact" href="#connect_plugindiscovery_migrateartifact">Operators: Artifact Migration</a></h5> + + <p>As an operator of Connect, if you discover incompatible plugins, there are multiple ways to try to resolve the incompatibility. They are listed below from most to least preferable.</p> + + <ol> + <li>Upgrade your incompatible plugins to the latest release version from your plugin provider.</li> + <li>Contact your plugin provider and request that they migrate the plugin to be compatible, following the <a href="#connect_plugindiscovery_migratesource">source migration instructions</a>, and then upgrade to the migrated version.</li> + <li>Migrate the plugin artifacts yourself using the included migration script.</li> + </ol> + + <p>The migration script is located in <code>bin/connect-plugin-path.sh</code> and <code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation. The script can migrate incompatible plugin artifacts already installed on your Connect worker's <code>plugin.path</code> by adding or modifying JAR or resource files. This is not suitable for environments using code-signing, as this may change the artifacts such that they will fail signature verification. View the built-in help with <code>--help</code>.</p> + + <p>To perform a migration, first use the <code>list</code> subcommand to get an overview of the plugins available to the script. You must tell the script where to find plugins, which can be done with the repeatable <code>--worker-config</code>, <code>--plugin-path</code>, and <code>--plugin-location</code> arguments. The script will only migrate plugins present in the paths specified, so if you add plugins to your worker's classpath, then you will need to specify those plugins via one or more <code>--plugin-location</code> arguments.</p> + + <p>Once you see that all incompatible plugins are included in the listing, you can proceed to dry-run the migration with <code>sync-manifests --dry-run</code>. This will perform all parts of the migration, except for writing the results of the migration to disk. Note that the <code>sync-manifests</code> command requires all specified paths to be writable, and may alter the contents of the directories. Make a backup of the specified paths, or copy them to a writable directory.</p> Review Comment: What does "Once you see that all incompatible plugins are included in the listing" mean? Does a user need to do more than just invoke the `connect-plugin-path list` command for this? If not, how about "If incompatible plugins are shown in the listing"? ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> Review Comment: It's a little unclear what "the other modes" are in "For version 3.6 and later, this mode... logs a warning for all plugins which are incompatible with the other modes." Maybe "logs a warning for all plugins that cannot be loaded with `SERVICE_LOAD` mode"? ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> + + <h5><a id="connect_plugindiscovery_migrateartifact" href="#connect_plugindiscovery_migrateartifact">Operators: Artifact Migration</a></h5> + + <p>As an operator of Connect, if you discover incompatible plugins, there are multiple ways to try to resolve the incompatibility. They are listed below from most to least preferable.</p> + + <ol> + <li>Upgrade your incompatible plugins to the latest release version from your plugin provider.</li> + <li>Contact your plugin provider and request that they migrate the plugin to be compatible, following the <a href="#connect_plugindiscovery_migratesource">source migration instructions</a>, and then upgrade to the migrated version.</li> + <li>Migrate the plugin artifacts yourself using the included migration script.</li> + </ol> + + <p>The migration script is located in <code>bin/connect-plugin-path.sh</code> and <code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation. The script can migrate incompatible plugin artifacts already installed on your Connect worker's <code>plugin.path</code> by adding or modifying JAR or resource files. This is not suitable for environments using code-signing, as this may change the artifacts such that they will fail signature verification. View the built-in help with <code>--help</code>.</p> + + <p>To perform a migration, first use the <code>list</code> subcommand to get an overview of the plugins available to the script. You must tell the script where to find plugins, which can be done with the repeatable <code>--worker-config</code>, <code>--plugin-path</code>, and <code>--plugin-location</code> arguments. The script will only migrate plugins present in the paths specified, so if you add plugins to your worker's classpath, then you will need to specify those plugins via one or more <code>--plugin-location</code> arguments.</p> + + <p>Once you see that all incompatible plugins are included in the listing, you can proceed to dry-run the migration with <code>sync-manifests --dry-run</code>. This will perform all parts of the migration, except for writing the results of the migration to disk. Note that the <code>sync-manifests</code> command requires all specified paths to be writable, and may alter the contents of the directories. Make a backup of the specified paths, or copy them to a writable directory.</p> + + <p>Ensure that you have a backup and the dry-run succeeds before removing the <code>--dry-run</code> flag and actually running the migration. If the migration fails without the <code>--dry-run</code> flag, then the partially migrated artifacts should be discarded. The migration is idempotent, so running it multiple times and on already-migrated plugins is safe. After the migration is completed, you should <a href="#connect_plugindiscovery_compatibility">verify the migration is complete</a>. These migration and verification steps may be put into a Continuous Integration process to automatically migrate plugins.</p> + + <h5><a id="connect_plugindiscovery_migratesource" href="#connect_plugindiscovery_migratesource">Developers: Source Migration</a></h5> + + <p>To make plugins compatible with <code>SERVICE_LOAD</code>, it is necessary to add <a href="https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html">ServiceLoader</a> manifests to your source code, which should then be packaged in the release artifact. Manifests are resource files in <code>META-INF/services/</code> named after their superclass type, and contain a list of fully-qualified subclass names, one on each line. Manifests should be located in the same module as the classes they refer to, not in the corresponding test resources.</p> + + <p>In order for a plugin to be compatible, it must appear as a line in a manifest corresponding to the plugin superclass it extends. If a single plugin implements multiple plugin interfaces, then it should appear in each corresponding manifest. If you have no classes for a certain type of plugin, you do not need to include a manifest file for that type. If you have classes which should not be visible as plugins, they should be marked abstract. The following types are expected to have manifests:</p> + + <ul> + <li><code>org.apache.kafka.connect.sink.SinkConnector</code></li> + <li><code>org.apache.kafka.connect.source.SourceConnector</code></li> + <li><code>org.apache.kafka.connect.storage.Converter</code></li> + <li><code>org.apache.kafka.connect.storage.HeaderConverter</code></li> + <li><code>org.apache.kafka.connect.transforms.Transformation</code></li> + <li><code>org.apache.kafka.connect.transforms.predicates.Predicate</code></li> + <li><code>org.apache.kafka.common.config.provider.ConfigProvider</code></li> + <li><code>org.apache.kafka.connect.rest.ConnectRestExtension</code></li> + <li><code>org.apache.kafka.connect.connector.policy.ConnectorClientConfigOverridePolicy</code></li> + </ul> + + <p>For example, if you only have one connector with the fully-qualified name <code>com.example.MySinkConnector</code>, then only one manifest file must be added to resources in <code>META-INF/services/org.apache.kafka.connect.sink.SinkConnector</code>, and the contents should be similar to the following:</p> + + <pre class="brush: resource;"> +# license header or comment +com.example.MySinkConnector</pre> + + <p>You should then verify that your manifests are correct. either with the <a href="#connect_plugindiscovery_compatibility">manual verification steps</a> or with unit tests. If you aren't using the <code>EmbeddedConnectCluster</code> you can add a test suite that starts up an <code>EmbeddedConnectCluster</code> and does nothing with it. If you are using the <code>EmbeddedConnectCluster</code>, then ensure you are using version 3.6 or later of the connect-runtime main and test artifacts. If you have a test that passes using the default <code>HYBRID_FAIL</code> strategy, then your migration is complete. You can then release the plugin normally, and operators can upgrade to the compatible version.</p> Review Comment: Nit: comma instead of period after "You should then verify that your manifests are correct" ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> Review Comment: Similar thought about "incompatible plugin" in "For unit-test environments... stops the worker with an error if an incompatible plugin is detected." Maybe "stops the worker if a plugin is detected that cannot be loaded with `SERVICE_LOAD` mode"? ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> Review Comment: Similar to to previous discussions about couching language regarding "unusable" plugins, could we change "Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable" to "Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and may make them unusable"? ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> Review Comment: Also, the formatting around [`plugin.discovery`](http://localhost:8080/documentation.html#connectconfigs_plugin.discovery)` worker configuration` is a little hard to read; IMO we can just do `<a href="#connectconfigs_plugin.discovery">plugin.discovery</a> worker configuration` and keep things simple without any `<code>` tags. ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> Review Comment: Are we being descriptive enough with the term "incompatible plugins" in "Finally, the `SERVICE_LOAD` strategy will silently hide incompatible plugins and make them unusable"? I know we go into more detail later, but if we're going to use that term so frequently, I think we should provide at least a brief summary of what that means in this section. Alternatively, we could say something like "Finally, the `SERVICE_LOAD` strategy will perform plugin scanning faster than other modes, but any plugins that have not been migrated to work with this mode will be silently hidden and may not be usable", which provides a tiny bit more detail and also reiterates the advantages of the `SERVICE_LOAD` mode. ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> Review Comment: Regarding "If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification" -- wouldn't the `connect-plugin-path` CLI tool be easier to use than either of these options? Shouldn't we provide that as an option for users to verify their Connect workers' setup with `SERVICE_LOAD` mode? ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> + + <h5><a id="connect_plugindiscovery_migrateartifact" href="#connect_plugindiscovery_migrateartifact">Operators: Artifact Migration</a></h5> + + <p>As an operator of Connect, if you discover incompatible plugins, there are multiple ways to try to resolve the incompatibility. They are listed below from most to least preferable.</p> + + <ol> + <li>Upgrade your incompatible plugins to the latest release version from your plugin provider.</li> + <li>Contact your plugin provider and request that they migrate the plugin to be compatible, following the <a href="#connect_plugindiscovery_migratesource">source migration instructions</a>, and then upgrade to the migrated version.</li> + <li>Migrate the plugin artifacts yourself using the included migration script.</li> + </ol> + + <p>The migration script is located in <code>bin/connect-plugin-path.sh</code> and <code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation. The script can migrate incompatible plugin artifacts already installed on your Connect worker's <code>plugin.path</code> by adding or modifying JAR or resource files. This is not suitable for environments using code-signing, as this may change the artifacts such that they will fail signature verification. View the built-in help with <code>--help</code>.</p> + + <p>To perform a migration, first use the <code>list</code> subcommand to get an overview of the plugins available to the script. You must tell the script where to find plugins, which can be done with the repeatable <code>--worker-config</code>, <code>--plugin-path</code>, and <code>--plugin-location</code> arguments. The script will only migrate plugins present in the paths specified, so if you add plugins to your worker's classpath, then you will need to specify those plugins via one or more <code>--plugin-location</code> arguments.</p> + + <p>Once you see that all incompatible plugins are included in the listing, you can proceed to dry-run the migration with <code>sync-manifests --dry-run</code>. This will perform all parts of the migration, except for writing the results of the migration to disk. Note that the <code>sync-manifests</code> command requires all specified paths to be writable, and may alter the contents of the directories. Make a backup of the specified paths, or copy them to a writable directory.</p> + + <p>Ensure that you have a backup and the dry-run succeeds before removing the <code>--dry-run</code> flag and actually running the migration. If the migration fails without the <code>--dry-run</code> flag, then the partially migrated artifacts should be discarded. The migration is idempotent, so running it multiple times and on already-migrated plugins is safe. After the migration is completed, you should <a href="#connect_plugindiscovery_compatibility">verify the migration is complete</a>. These migration and verification steps may be put into a Continuous Integration process to automatically migrate plugins.</p> + + <h5><a id="connect_plugindiscovery_migratesource" href="#connect_plugindiscovery_migratesource">Developers: Source Migration</a></h5> + + <p>To make plugins compatible with <code>SERVICE_LOAD</code>, it is necessary to add <a href="https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html">ServiceLoader</a> manifests to your source code, which should then be packaged in the release artifact. Manifests are resource files in <code>META-INF/services/</code> named after their superclass type, and contain a list of fully-qualified subclass names, one on each line. Manifests should be located in the same module as the classes they refer to, not in the corresponding test resources.</p> + + <p>In order for a plugin to be compatible, it must appear as a line in a manifest corresponding to the plugin superclass it extends. If a single plugin implements multiple plugin interfaces, then it should appear in each corresponding manifest. If you have no classes for a certain type of plugin, you do not need to include a manifest file for that type. If you have classes which should not be visible as plugins, they should be marked abstract. The following types are expected to have manifests:</p> + + <ul> + <li><code>org.apache.kafka.connect.sink.SinkConnector</code></li> + <li><code>org.apache.kafka.connect.source.SourceConnector</code></li> + <li><code>org.apache.kafka.connect.storage.Converter</code></li> + <li><code>org.apache.kafka.connect.storage.HeaderConverter</code></li> + <li><code>org.apache.kafka.connect.transforms.Transformation</code></li> + <li><code>org.apache.kafka.connect.transforms.predicates.Predicate</code></li> + <li><code>org.apache.kafka.common.config.provider.ConfigProvider</code></li> + <li><code>org.apache.kafka.connect.rest.ConnectRestExtension</code></li> + <li><code>org.apache.kafka.connect.connector.policy.ConnectorClientConfigOverridePolicy</code></li> + </ul> + + <p>For example, if you only have one connector with the fully-qualified name <code>com.example.MySinkConnector</code>, then only one manifest file must be added to resources in <code>META-INF/services/org.apache.kafka.connect.sink.SinkConnector</code>, and the contents should be similar to the following:</p> + + <pre class="brush: resource;"> +# license header or comment +com.example.MySinkConnector</pre> + + <p>You should then verify that your manifests are correct. either with the <a href="#connect_plugindiscovery_compatibility">manual verification steps</a> or with unit tests. If you aren't using the <code>EmbeddedConnectCluster</code> you can add a test suite that starts up an <code>EmbeddedConnectCluster</code> and does nothing with it. If you are using the <code>EmbeddedConnectCluster</code>, then ensure you are using version 3.6 or later of the connect-runtime main and test artifacts. If you have a test that passes using the default <code>HYBRID_FAIL</code> strategy, then your migration is complete. You can then release the plugin normally, and operators can upgrade to the compatible version.</p> Review Comment: Also, same thought RE explicit mention of the `EmbeddedConnectCluster` class. The instructions on starting a worker in a testing environment with `HYBRID_FAIL` mode are great, we should just be a tad more general about how we describe that kind of setup. This is also useful since some users have their own integration testing environments which don't use the `EmbeddedConnectCluster` class but can still use the `HYBRID_FAIL` testing strategy. ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> + + <h5><a id="connect_plugindiscovery_migrateartifact" href="#connect_plugindiscovery_migrateartifact">Operators: Artifact Migration</a></h5> + + <p>As an operator of Connect, if you discover incompatible plugins, there are multiple ways to try to resolve the incompatibility. They are listed below from most to least preferable.</p> + + <ol> + <li>Upgrade your incompatible plugins to the latest release version from your plugin provider.</li> + <li>Contact your plugin provider and request that they migrate the plugin to be compatible, following the <a href="#connect_plugindiscovery_migratesource">source migration instructions</a>, and then upgrade to the migrated version.</li> + <li>Migrate the plugin artifacts yourself using the included migration script.</li> + </ol> + + <p>The migration script is located in <code>bin/connect-plugin-path.sh</code> and <code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation. The script can migrate incompatible plugin artifacts already installed on your Connect worker's <code>plugin.path</code> by adding or modifying JAR or resource files. This is not suitable for environments using code-signing, as this may change the artifacts such that they will fail signature verification. View the built-in help with <code>--help</code>.</p> + + <p>To perform a migration, first use the <code>list</code> subcommand to get an overview of the plugins available to the script. You must tell the script where to find plugins, which can be done with the repeatable <code>--worker-config</code>, <code>--plugin-path</code>, and <code>--plugin-location</code> arguments. The script will only migrate plugins present in the paths specified, so if you add plugins to your worker's classpath, then you will need to specify those plugins via one or more <code>--plugin-location</code> arguments.</p> + + <p>Once you see that all incompatible plugins are included in the listing, you can proceed to dry-run the migration with <code>sync-manifests --dry-run</code>. This will perform all parts of the migration, except for writing the results of the migration to disk. Note that the <code>sync-manifests</code> command requires all specified paths to be writable, and may alter the contents of the directories. Make a backup of the specified paths, or copy them to a writable directory.</p> + + <p>Ensure that you have a backup and the dry-run succeeds before removing the <code>--dry-run</code> flag and actually running the migration. If the migration fails without the <code>--dry-run</code> flag, then the partially migrated artifacts should be discarded. The migration is idempotent, so running it multiple times and on already-migrated plugins is safe. After the migration is completed, you should <a href="#connect_plugindiscovery_compatibility">verify the migration is complete</a>. These migration and verification steps may be put into a Continuous Integration process to automatically migrate plugins.</p> + + <h5><a id="connect_plugindiscovery_migratesource" href="#connect_plugindiscovery_migratesource">Developers: Source Migration</a></h5> + + <p>To make plugins compatible with <code>SERVICE_LOAD</code>, it is necessary to add <a href="https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html">ServiceLoader</a> manifests to your source code, which should then be packaged in the release artifact. Manifests are resource files in <code>META-INF/services/</code> named after their superclass type, and contain a list of fully-qualified subclass names, one on each line. Manifests should be located in the same module as the classes they refer to, not in the corresponding test resources.</p> + + <p>In order for a plugin to be compatible, it must appear as a line in a manifest corresponding to the plugin superclass it extends. If a single plugin implements multiple plugin interfaces, then it should appear in each corresponding manifest. If you have no classes for a certain type of plugin, you do not need to include a manifest file for that type. If you have classes which should not be visible as plugins, they should be marked abstract. The following types are expected to have manifests:</p> Review Comment: Nit: maybe instead of "If a single plugin implements multiple plugin interfaces, then it should appear in each corresponding manifest" we can say "If a single plugin implements multiple plugin interfaces, then it should appear in a manifest for each plugin interface that it implements"? ########## docs/connect.html: ########## @@ -583,6 +644,10 @@ <h5><a id="connect_connectorexample" href="#connect_connectorexample">Connector public class FileStreamSourceConnector extends SourceConnector { private Map<String, String> props;</pre> + <p>Add a corresponding ServiceLoader manifest to your resources in <code>META-INF/services/org.apache.kafka.connect.source.SourceConnector</code> to allow it to be discovered by the runtime.</p> Review Comment: This kind of breaks the flow of writing our connector class; it comes between the `public class FileStreamConnector...` declaration and the `public Class<? extends Task> taskClass()` declaration. Maybe we can put this before the class declaration, with a note that the name of the connector we're developing will be `FileStreamSourceConnector`? Also, we should probably link to the docs on plugin discovery modes in this section. Maybe something like "See <a href="#connect_plugindiscovery">plugin discovery</a> for more info on this step"? ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> + + <ul> + <li>Start your worker with the default <code>HYBRID_WARN</code>strategy, and WARN logs enabled for the <code>org.apache.kafka.connect</code> package. At least one WARN log message mentioning the <code>plugin.discovery</code> configuration should be printed. This log message will explicitly say that all plugins are compatible, or list the incompatible plugins.</li> + <li>Start your worker in a test environment with <code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed. If at least one plugin is not compatible the worker will fail to start up, and all incompatible plugins will be listed in the exception.</li> + </ul> + + <p>If the verification step succeeds, then your current set of installed plugins are compatible, and it should be safe to change the <code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If you change the set of already-installed plugins, they may no longer be compatible, and you should repeat the above verification. If the verification fails, you must address the incompatible plugins before using the <code>SERVICE_LOAD</code> strategy.</p> + + <h5><a id="connect_plugindiscovery_migrateartifact" href="#connect_plugindiscovery_migrateartifact">Operators: Artifact Migration</a></h5> + + <p>As an operator of Connect, if you discover incompatible plugins, there are multiple ways to try to resolve the incompatibility. They are listed below from most to least preferable.</p> + + <ol> + <li>Upgrade your incompatible plugins to the latest release version from your plugin provider.</li> + <li>Contact your plugin provider and request that they migrate the plugin to be compatible, following the <a href="#connect_plugindiscovery_migratesource">source migration instructions</a>, and then upgrade to the migrated version.</li> + <li>Migrate the plugin artifacts yourself using the included migration script.</li> + </ol> + + <p>The migration script is located in <code>bin/connect-plugin-path.sh</code> and <code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation. The script can migrate incompatible plugin artifacts already installed on your Connect worker's <code>plugin.path</code> by adding or modifying JAR or resource files. This is not suitable for environments using code-signing, as this may change the artifacts such that they will fail signature verification. View the built-in help with <code>--help</code>.</p> + + <p>To perform a migration, first use the <code>list</code> subcommand to get an overview of the plugins available to the script. You must tell the script where to find plugins, which can be done with the repeatable <code>--worker-config</code>, <code>--plugin-path</code>, and <code>--plugin-location</code> arguments. The script will only migrate plugins present in the paths specified, so if you add plugins to your worker's classpath, then you will need to specify those plugins via one or more <code>--plugin-location</code> arguments.</p> + + <p>Once you see that all incompatible plugins are included in the listing, you can proceed to dry-run the migration with <code>sync-manifests --dry-run</code>. This will perform all parts of the migration, except for writing the results of the migration to disk. Note that the <code>sync-manifests</code> command requires all specified paths to be writable, and may alter the contents of the directories. Make a backup of the specified paths, or copy them to a writable directory.</p> + + <p>Ensure that you have a backup and the dry-run succeeds before removing the <code>--dry-run</code> flag and actually running the migration. If the migration fails without the <code>--dry-run</code> flag, then the partially migrated artifacts should be discarded. The migration is idempotent, so running it multiple times and on already-migrated plugins is safe. After the migration is completed, you should <a href="#connect_plugindiscovery_compatibility">verify the migration is complete</a>. These migration and verification steps may be put into a Continuous Integration process to automatically migrate plugins.</p> + + <h5><a id="connect_plugindiscovery_migratesource" href="#connect_plugindiscovery_migratesource">Developers: Source Migration</a></h5> + + <p>To make plugins compatible with <code>SERVICE_LOAD</code>, it is necessary to add <a href="https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html">ServiceLoader</a> manifests to your source code, which should then be packaged in the release artifact. Manifests are resource files in <code>META-INF/services/</code> named after their superclass type, and contain a list of fully-qualified subclass names, one on each line. Manifests should be located in the same module as the classes they refer to, not in the corresponding test resources.</p> Review Comment: Is the "Manifests should be located in the same module as the classes they refer to, not in the corresponding test resources." part necessary? When do we expect users to do something like this? ########## docs/connect.html: ########## @@ -543,6 +543,67 @@ <h6>ACL requirements</h6> </tbody> </table> + <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin Discovery</a></h4> + + <p>Plugin discovery is the name for the strategy which the Connect worker uses to find plugin classes and make them accessible to configure and run in Connectors and Tasks. This is controlled by the <a href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker configuration</a>, and has a significant impact on worker startup time. <code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to verify that plugins are compatible before setting this configuration to <code>SERVICE_LOAD</code>.</p> + + <p>Prior to version 3.6, this strategy was not configurable, and behaved like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is also compatible with all plugins, but logs a warning for all plugins which are incompatible with the other modes. For unit-test environments that use the <code>EmbeddedConnectCluster</code> this defaults to the <code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code> strategy will silently hide incompatible plugins and make them unusable.</p> + + <h5><a id="connect_plugindiscovery_compatibility" href="#connect_plugindiscovery_compatibility">Verifying Plugin Compatibility</a></h5> + + <p>To verify if all of your plugins are compatible, first ensure that you are using version 3.6 or later of the Connect runtime. You can then perform one of the following checks:</p> Review Comment: Can just say "Kafka Connect" instead of "the Connect runtime". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org