C0urante commented on code in PR #14068:
URL: https://github.com/apache/kafka/pull/14068#discussion_r1283295501
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
Review Comment:
I don't think we should explicitly call out the `EmbeddedConnectCluster`
class, since it's not public API and discussing it in our docs may mistakenly
give people the impression that it is.
Perhaps we could mention the general utility of the `HYBRID_FAIL` mode in
testing environments without providing specific details of how we use it
ourselves?
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
Review Comment:
Does "Connectors and Tasks" cover anything that just "connectors" doesn't?
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
Review Comment:
Can be more explicit: "compatible with `SERVICE_LOAD` mode" instead of just
"compatible.
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
+
+ <ul>
+ <li>Start your worker with the default
<code>HYBRID_WARN</code>strategy, and WARN logs enabled for the
<code>org.apache.kafka.connect</code> package. At least one WARN log message
mentioning the <code>plugin.discovery</code> configuration should be printed.
This log message will explicitly say that all plugins are compatible, or list
the incompatible plugins.</li>
+ <li>Start your worker in a test environment with
<code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed.
If at least one plugin is not compatible the worker will fail to start up, and
all incompatible plugins will be listed in the exception.</li>
+ </ul>
+
+ <p>If the verification step succeeds, then your current set of installed
plugins are compatible, and it should be safe to change the
<code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If
you change the set of already-installed plugins, they may no longer be
compatible, and you should repeat the above verification. If the verification
fails, you must address the incompatible plugins before using the
<code>SERVICE_LOAD</code> strategy.</p>
+
+ <h5><a id="connect_plugindiscovery_migrateartifact"
href="#connect_plugindiscovery_migrateartifact">Operators: Artifact
Migration</a></h5>
+
+ <p>As an operator of Connect, if you discover incompatible plugins, there
are multiple ways to try to resolve the incompatibility. They are listed below
from most to least preferable.</p>
+
+ <ol>
+ <li>Upgrade your incompatible plugins to the latest release version
from your plugin provider.</li>
+ <li>Contact your plugin provider and request that they migrate the
plugin to be compatible, following the <a
href="#connect_plugindiscovery_migratesource">source migration
instructions</a>, and then upgrade to the migrated version.</li>
+ <li>Migrate the plugin artifacts yourself using the included migration
script.</li>
+ </ol>
+
+ <p>The migration script is located in
<code>bin/connect-plugin-path.sh</code> and
<code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation.
The script can migrate incompatible plugin artifacts already installed on your
Connect worker's <code>plugin.path</code> by adding or modifying JAR or
resource files. This is not suitable for environments using code-signing, as
this may change the artifacts such that they will fail signature verification.
View the built-in help with <code>--help</code>.</p>
+
+ <p>To perform a migration, first use the <code>list</code> subcommand to
get an overview of the plugins available to the script. You must tell the
script where to find plugins, which can be done with the repeatable
<code>--worker-config</code>, <code>--plugin-path</code>, and
<code>--plugin-location</code> arguments. The script will only migrate plugins
present in the paths specified, so if you add plugins to your worker's
classpath, then you will need to specify those plugins via one or more
<code>--plugin-location</code> arguments.</p>
Review Comment:
This part seems a little unclear: "The script will only migrate plugins
present in the paths specified, so if you add plugins to your worker's
classpath, then you will need to specify those plugins via one or more
<code>--plugin-location</code> arguments."
Maybe "Note that the script will ignore plugins present on the Connect
worker's classpath unless they are contained in an explicitly-provided
<code>--plugin-location</code>"?
Also, can you refresh my memory on why we've designed the script to ignore
classpath plugins? It feels like this has made things more difficult, both for
us (implementing the `connect-plugin-path` script) and users (adding a footgun
for plugins installed directly on the classpath). IMO it's not too late to
tweak this behavior if we decide it's better to take things in a different
direction.
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
+
+ <ul>
+ <li>Start your worker with the default
<code>HYBRID_WARN</code>strategy, and WARN logs enabled for the
<code>org.apache.kafka.connect</code> package. At least one WARN log message
mentioning the <code>plugin.discovery</code> configuration should be printed.
This log message will explicitly say that all plugins are compatible, or list
the incompatible plugins.</li>
+ <li>Start your worker in a test environment with
<code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed.
If at least one plugin is not compatible the worker will fail to start up, and
all incompatible plugins will be listed in the exception.</li>
+ </ul>
+
+ <p>If the verification step succeeds, then your current set of installed
plugins are compatible, and it should be safe to change the
<code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If
you change the set of already-installed plugins, they may no longer be
compatible, and you should repeat the above verification. If the verification
fails, you must address the incompatible plugins before using the
<code>SERVICE_LOAD</code> strategy.</p>
Review Comment:
Nit: "set of installed plugins is compatible" instead of "set of installed
plugins are compatible"
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
+
+ <ul>
+ <li>Start your worker with the default
<code>HYBRID_WARN</code>strategy, and WARN logs enabled for the
<code>org.apache.kafka.connect</code> package. At least one WARN log message
mentioning the <code>plugin.discovery</code> configuration should be printed.
This log message will explicitly say that all plugins are compatible, or list
the incompatible plugins.</li>
+ <li>Start your worker in a test environment with
<code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed.
If at least one plugin is not compatible the worker will fail to start up, and
all incompatible plugins will be listed in the exception.</li>
+ </ul>
+
+ <p>If the verification step succeeds, then your current set of installed
plugins are compatible, and it should be safe to change the
<code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If
you change the set of already-installed plugins, they may no longer be
compatible, and you should repeat the above verification. If the verification
fails, you must address the incompatible plugins before using the
<code>SERVICE_LOAD</code> strategy.</p>
+
+ <h5><a id="connect_plugindiscovery_migrateartifact"
href="#connect_plugindiscovery_migrateartifact">Operators: Artifact
Migration</a></h5>
+
+ <p>As an operator of Connect, if you discover incompatible plugins, there
are multiple ways to try to resolve the incompatibility. They are listed below
from most to least preferable.</p>
+
+ <ol>
+ <li>Upgrade your incompatible plugins to the latest release version
from your plugin provider.</li>
Review Comment:
This is a little misleading; someone may stop reading here and assume that
an upgrade is guaranteed to fix the problem.
Maybe "Check the latest release of your plugin and, if it is compatible,
upgrade to that version"?
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
+
+ <ul>
+ <li>Start your worker with the default
<code>HYBRID_WARN</code>strategy, and WARN logs enabled for the
<code>org.apache.kafka.connect</code> package. At least one WARN log message
mentioning the <code>plugin.discovery</code> configuration should be printed.
This log message will explicitly say that all plugins are compatible, or list
the incompatible plugins.</li>
+ <li>Start your worker in a test environment with
<code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed.
If at least one plugin is not compatible the worker will fail to start up, and
all incompatible plugins will be listed in the exception.</li>
+ </ul>
+
+ <p>If the verification step succeeds, then your current set of installed
plugins are compatible, and it should be safe to change the
<code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If
you change the set of already-installed plugins, they may no longer be
compatible, and you should repeat the above verification. If the verification
fails, you must address the incompatible plugins before using the
<code>SERVICE_LOAD</code> strategy.</p>
+
+ <h5><a id="connect_plugindiscovery_migrateartifact"
href="#connect_plugindiscovery_migrateartifact">Operators: Artifact
Migration</a></h5>
+
+ <p>As an operator of Connect, if you discover incompatible plugins, there
are multiple ways to try to resolve the incompatibility. They are listed below
from most to least preferable.</p>
+
+ <ol>
+ <li>Upgrade your incompatible plugins to the latest release version
from your plugin provider.</li>
+ <li>Contact your plugin provider and request that they migrate the
plugin to be compatible, following the <a
href="#connect_plugindiscovery_migratesource">source migration
instructions</a>, and then upgrade to the migrated version.</li>
+ <li>Migrate the plugin artifacts yourself using the included migration
script.</li>
+ </ol>
+
+ <p>The migration script is located in
<code>bin/connect-plugin-path.sh</code> and
<code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation.
The script can migrate incompatible plugin artifacts already installed on your
Connect worker's <code>plugin.path</code> by adding or modifying JAR or
resource files. This is not suitable for environments using code-signing, as
this may change the artifacts such that they will fail signature verification.
View the built-in help with <code>--help</code>.</p>
+
+ <p>To perform a migration, first use the <code>list</code> subcommand to
get an overview of the plugins available to the script. You must tell the
script where to find plugins, which can be done with the repeatable
<code>--worker-config</code>, <code>--plugin-path</code>, and
<code>--plugin-location</code> arguments. The script will only migrate plugins
present in the paths specified, so if you add plugins to your worker's
classpath, then you will need to specify those plugins via one or more
<code>--plugin-location</code> arguments.</p>
+
+ <p>Once you see that all incompatible plugins are included in the listing,
you can proceed to dry-run the migration with <code>sync-manifests
--dry-run</code>. This will perform all parts of the migration, except for
writing the results of the migration to disk. Note that the
<code>sync-manifests</code> command requires all specified paths to be
writable, and may alter the contents of the directories. Make a backup of the
specified paths, or copy them to a writable directory.</p>
+
+ <p>Ensure that you have a backup and the dry-run succeeds before removing
the <code>--dry-run</code> flag and actually running the migration. If the
migration fails without the <code>--dry-run</code> flag, then the partially
migrated artifacts should be discarded. The migration is idempotent, so running
it multiple times and on already-migrated plugins is safe. After the migration
is completed, you should <a
href="#connect_plugindiscovery_compatibility">verify the migration is
complete</a>. These migration and verification steps may be put into a
Continuous Integration process to automatically migrate plugins.</p>
Review Comment:
In "Ensure that you have a backup", it's a little unclear what "backup"
refers to. Perhaps "Ensure that you have a backup of your plugins"?
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
+
+ <ul>
+ <li>Start your worker with the default
<code>HYBRID_WARN</code>strategy, and WARN logs enabled for the
<code>org.apache.kafka.connect</code> package. At least one WARN log message
mentioning the <code>plugin.discovery</code> configuration should be printed.
This log message will explicitly say that all plugins are compatible, or list
the incompatible plugins.</li>
+ <li>Start your worker in a test environment with
<code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed.
If at least one plugin is not compatible the worker will fail to start up, and
all incompatible plugins will be listed in the exception.</li>
+ </ul>
+
+ <p>If the verification step succeeds, then your current set of installed
plugins are compatible, and it should be safe to change the
<code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If
you change the set of already-installed plugins, they may no longer be
compatible, and you should repeat the above verification. If the verification
fails, you must address the incompatible plugins before using the
<code>SERVICE_LOAD</code> strategy.</p>
+
+ <h5><a id="connect_plugindiscovery_migrateartifact"
href="#connect_plugindiscovery_migrateartifact">Operators: Artifact
Migration</a></h5>
+
+ <p>As an operator of Connect, if you discover incompatible plugins, there
are multiple ways to try to resolve the incompatibility. They are listed below
from most to least preferable.</p>
+
+ <ol>
+ <li>Upgrade your incompatible plugins to the latest release version
from your plugin provider.</li>
+ <li>Contact your plugin provider and request that they migrate the
plugin to be compatible, following the <a
href="#connect_plugindiscovery_migratesource">source migration
instructions</a>, and then upgrade to the migrated version.</li>
+ <li>Migrate the plugin artifacts yourself using the included migration
script.</li>
+ </ol>
+
+ <p>The migration script is located in
<code>bin/connect-plugin-path.sh</code> and
<code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation.
The script can migrate incompatible plugin artifacts already installed on your
Connect worker's <code>plugin.path</code> by adding or modifying JAR or
resource files. This is not suitable for environments using code-signing, as
this may change the artifacts such that they will fail signature verification.
View the built-in help with <code>--help</code>.</p>
+
+ <p>To perform a migration, first use the <code>list</code> subcommand to
get an overview of the plugins available to the script. You must tell the
script where to find plugins, which can be done with the repeatable
<code>--worker-config</code>, <code>--plugin-path</code>, and
<code>--plugin-location</code> arguments. The script will only migrate plugins
present in the paths specified, so if you add plugins to your worker's
classpath, then you will need to specify those plugins via one or more
<code>--plugin-location</code> arguments.</p>
+
+ <p>Once you see that all incompatible plugins are included in the listing,
you can proceed to dry-run the migration with <code>sync-manifests
--dry-run</code>. This will perform all parts of the migration, except for
writing the results of the migration to disk. Note that the
<code>sync-manifests</code> command requires all specified paths to be
writable, and may alter the contents of the directories. Make a backup of the
specified paths, or copy them to a writable directory.</p>
Review Comment:
What does "Once you see that all incompatible plugins are included in the
listing" mean? Does a user need to do more than just invoke the
`connect-plugin-path list` command for this? If not, how about "If incompatible
plugins are shown in the listing"?
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
Review Comment:
It's a little unclear what "the other modes" are in "For version 3.6 and
later, this mode... logs a warning for all plugins which are incompatible with
the other modes."
Maybe "logs a warning for all plugins that cannot be loaded with
`SERVICE_LOAD` mode"?
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
+
+ <ul>
+ <li>Start your worker with the default
<code>HYBRID_WARN</code>strategy, and WARN logs enabled for the
<code>org.apache.kafka.connect</code> package. At least one WARN log message
mentioning the <code>plugin.discovery</code> configuration should be printed.
This log message will explicitly say that all plugins are compatible, or list
the incompatible plugins.</li>
+ <li>Start your worker in a test environment with
<code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed.
If at least one plugin is not compatible the worker will fail to start up, and
all incompatible plugins will be listed in the exception.</li>
+ </ul>
+
+ <p>If the verification step succeeds, then your current set of installed
plugins are compatible, and it should be safe to change the
<code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If
you change the set of already-installed plugins, they may no longer be
compatible, and you should repeat the above verification. If the verification
fails, you must address the incompatible plugins before using the
<code>SERVICE_LOAD</code> strategy.</p>
+
+ <h5><a id="connect_plugindiscovery_migrateartifact"
href="#connect_plugindiscovery_migrateartifact">Operators: Artifact
Migration</a></h5>
+
+ <p>As an operator of Connect, if you discover incompatible plugins, there
are multiple ways to try to resolve the incompatibility. They are listed below
from most to least preferable.</p>
+
+ <ol>
+ <li>Upgrade your incompatible plugins to the latest release version
from your plugin provider.</li>
+ <li>Contact your plugin provider and request that they migrate the
plugin to be compatible, following the <a
href="#connect_plugindiscovery_migratesource">source migration
instructions</a>, and then upgrade to the migrated version.</li>
+ <li>Migrate the plugin artifacts yourself using the included migration
script.</li>
+ </ol>
+
+ <p>The migration script is located in
<code>bin/connect-plugin-path.sh</code> and
<code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation.
The script can migrate incompatible plugin artifacts already installed on your
Connect worker's <code>plugin.path</code> by adding or modifying JAR or
resource files. This is not suitable for environments using code-signing, as
this may change the artifacts such that they will fail signature verification.
View the built-in help with <code>--help</code>.</p>
+
+ <p>To perform a migration, first use the <code>list</code> subcommand to
get an overview of the plugins available to the script. You must tell the
script where to find plugins, which can be done with the repeatable
<code>--worker-config</code>, <code>--plugin-path</code>, and
<code>--plugin-location</code> arguments. The script will only migrate plugins
present in the paths specified, so if you add plugins to your worker's
classpath, then you will need to specify those plugins via one or more
<code>--plugin-location</code> arguments.</p>
+
+ <p>Once you see that all incompatible plugins are included in the listing,
you can proceed to dry-run the migration with <code>sync-manifests
--dry-run</code>. This will perform all parts of the migration, except for
writing the results of the migration to disk. Note that the
<code>sync-manifests</code> command requires all specified paths to be
writable, and may alter the contents of the directories. Make a backup of the
specified paths, or copy them to a writable directory.</p>
+
+ <p>Ensure that you have a backup and the dry-run succeeds before removing
the <code>--dry-run</code> flag and actually running the migration. If the
migration fails without the <code>--dry-run</code> flag, then the partially
migrated artifacts should be discarded. The migration is idempotent, so running
it multiple times and on already-migrated plugins is safe. After the migration
is completed, you should <a
href="#connect_plugindiscovery_compatibility">verify the migration is
complete</a>. These migration and verification steps may be put into a
Continuous Integration process to automatically migrate plugins.</p>
+
+ <h5><a id="connect_plugindiscovery_migratesource"
href="#connect_plugindiscovery_migratesource">Developers: Source
Migration</a></h5>
+
+ <p>To make plugins compatible with <code>SERVICE_LOAD</code>, it is
necessary to add <a
href="https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html";>ServiceLoader</a>
manifests to your source code, which should then be packaged in the release
artifact. Manifests are resource files in <code>META-INF/services/</code> named
after their superclass type, and contain a list of fully-qualified subclass
names, one on each line. Manifests should be located in the same module as the
classes they refer to, not in the corresponding test resources.</p>
+
+ <p>In order for a plugin to be compatible, it must appear as a line in a
manifest corresponding to the plugin superclass it extends. If a single plugin
implements multiple plugin interfaces, then it should appear in each
corresponding manifest. If you have no classes for a certain type of plugin,
you do not need to include a manifest file for that type. If you have classes
which should not be visible as plugins, they should be marked abstract. The
following types are expected to have manifests:</p>
+
+ <ul>
+ <li><code>org.apache.kafka.connect.sink.SinkConnector</code></li>
+ <li><code>org.apache.kafka.connect.source.SourceConnector</code></li>
+ <li><code>org.apache.kafka.connect.storage.Converter</code></li>
+ <li><code>org.apache.kafka.connect.storage.HeaderConverter</code></li>
+
<li><code>org.apache.kafka.connect.transforms.Transformation</code></li>
+
<li><code>org.apache.kafka.connect.transforms.predicates.Predicate</code></li>
+
<li><code>org.apache.kafka.common.config.provider.ConfigProvider</code></li>
+
<li><code>org.apache.kafka.connect.rest.ConnectRestExtension</code></li>
+
<li><code>org.apache.kafka.connect.connector.policy.ConnectorClientConfigOverridePolicy</code></li>
+ </ul>
+
+ <p>For example, if you only have one connector with the fully-qualified
name <code>com.example.MySinkConnector</code>, then only one manifest file must
be added to resources in
<code>META-INF/services/org.apache.kafka.connect.sink.SinkConnector</code>, and
the contents should be similar to the following:</p>
+
+ <pre class="brush: resource;">
+# license header or comment
+com.example.MySinkConnector</pre>
+
+ <p>You should then verify that your manifests are correct. either with the
<a href="#connect_plugindiscovery_compatibility">manual verification steps</a>
or with unit tests. If you aren't using the <code>EmbeddedConnectCluster</code>
you can add a test suite that starts up an <code>EmbeddedConnectCluster</code>
and does nothing with it. If you are using the
<code>EmbeddedConnectCluster</code>, then ensure you are using version 3.6 or
later of the connect-runtime main and test artifacts. If you have a test that
passes using the default <code>HYBRID_FAIL</code> strategy, then your migration
is complete. You can then release the plugin normally, and operators can
upgrade to the compatible version.</p>
Review Comment:
Nit: comma instead of period after "You should then verify that your
manifests are correct"
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
Review Comment:
Similar thought about "incompatible plugin" in "For unit-test
environments... stops the worker with an error if an incompatible plugin is
detected."
Maybe "stops the worker if a plugin is detected that cannot be loaded with
`SERVICE_LOAD` mode"?
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
Review Comment:
Similar to to previous discussions about couching language regarding
"unusable" plugins, could we change "Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable" to
"Finally, the <code>SERVICE_LOAD</code> strategy will silently hide
incompatible plugins and may make them unusable"?
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
Review Comment:
Also, the formatting around
[`plugin.discovery`](http://localhost:8080/documentation.html#connectconfigs_plugin.discovery)`
worker configuration` is a little hard to read; IMO we can just do `<a
href="#connectconfigs_plugin.discovery">plugin.discovery</a> worker
configuration` and keep things simple without any `<code>` tags.
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
Review Comment:
Are we being descriptive enough with the term "incompatible plugins" in
"Finally, the `SERVICE_LOAD` strategy will silently hide incompatible plugins
and make them unusable"? I know we go into more detail later, but if we're
going to use that term so frequently, I think we should provide at least a
brief summary of what that means in this section. Alternatively, we could say
something like "Finally, the `SERVICE_LOAD` strategy will perform plugin
scanning faster than other modes, but any plugins that have not been migrated
to work with this mode will be silently hidden and may not be usable", which
provides a tiny bit more detail and also reiterates the advantages of the
`SERVICE_LOAD` mode.
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
+
+ <ul>
+ <li>Start your worker with the default
<code>HYBRID_WARN</code>strategy, and WARN logs enabled for the
<code>org.apache.kafka.connect</code> package. At least one WARN log message
mentioning the <code>plugin.discovery</code> configuration should be printed.
This log message will explicitly say that all plugins are compatible, or list
the incompatible plugins.</li>
+ <li>Start your worker in a test environment with
<code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed.
If at least one plugin is not compatible the worker will fail to start up, and
all incompatible plugins will be listed in the exception.</li>
+ </ul>
+
+ <p>If the verification step succeeds, then your current set of installed
plugins are compatible, and it should be safe to change the
<code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If
you change the set of already-installed plugins, they may no longer be
compatible, and you should repeat the above verification. If the verification
fails, you must address the incompatible plugins before using the
<code>SERVICE_LOAD</code> strategy.</p>
Review Comment:
Regarding "If you change the set of already-installed plugins, they may no
longer be compatible, and you should repeat the above verification" -- wouldn't
the `connect-plugin-path` CLI tool be easier to use than either of these
options? Shouldn't we provide that as an option for users to verify their
Connect workers' setup with `SERVICE_LOAD` mode?
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
+
+ <ul>
+ <li>Start your worker with the default
<code>HYBRID_WARN</code>strategy, and WARN logs enabled for the
<code>org.apache.kafka.connect</code> package. At least one WARN log message
mentioning the <code>plugin.discovery</code> configuration should be printed.
This log message will explicitly say that all plugins are compatible, or list
the incompatible plugins.</li>
+ <li>Start your worker in a test environment with
<code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed.
If at least one plugin is not compatible the worker will fail to start up, and
all incompatible plugins will be listed in the exception.</li>
+ </ul>
+
+ <p>If the verification step succeeds, then your current set of installed
plugins are compatible, and it should be safe to change the
<code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If
you change the set of already-installed plugins, they may no longer be
compatible, and you should repeat the above verification. If the verification
fails, you must address the incompatible plugins before using the
<code>SERVICE_LOAD</code> strategy.</p>
+
+ <h5><a id="connect_plugindiscovery_migrateartifact"
href="#connect_plugindiscovery_migrateartifact">Operators: Artifact
Migration</a></h5>
+
+ <p>As an operator of Connect, if you discover incompatible plugins, there
are multiple ways to try to resolve the incompatibility. They are listed below
from most to least preferable.</p>
+
+ <ol>
+ <li>Upgrade your incompatible plugins to the latest release version
from your plugin provider.</li>
+ <li>Contact your plugin provider and request that they migrate the
plugin to be compatible, following the <a
href="#connect_plugindiscovery_migratesource">source migration
instructions</a>, and then upgrade to the migrated version.</li>
+ <li>Migrate the plugin artifacts yourself using the included migration
script.</li>
+ </ol>
+
+ <p>The migration script is located in
<code>bin/connect-plugin-path.sh</code> and
<code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation.
The script can migrate incompatible plugin artifacts already installed on your
Connect worker's <code>plugin.path</code> by adding or modifying JAR or
resource files. This is not suitable for environments using code-signing, as
this may change the artifacts such that they will fail signature verification.
View the built-in help with <code>--help</code>.</p>
+
+ <p>To perform a migration, first use the <code>list</code> subcommand to
get an overview of the plugins available to the script. You must tell the
script where to find plugins, which can be done with the repeatable
<code>--worker-config</code>, <code>--plugin-path</code>, and
<code>--plugin-location</code> arguments. The script will only migrate plugins
present in the paths specified, so if you add plugins to your worker's
classpath, then you will need to specify those plugins via one or more
<code>--plugin-location</code> arguments.</p>
+
+ <p>Once you see that all incompatible plugins are included in the listing,
you can proceed to dry-run the migration with <code>sync-manifests
--dry-run</code>. This will perform all parts of the migration, except for
writing the results of the migration to disk. Note that the
<code>sync-manifests</code> command requires all specified paths to be
writable, and may alter the contents of the directories. Make a backup of the
specified paths, or copy them to a writable directory.</p>
+
+ <p>Ensure that you have a backup and the dry-run succeeds before removing
the <code>--dry-run</code> flag and actually running the migration. If the
migration fails without the <code>--dry-run</code> flag, then the partially
migrated artifacts should be discarded. The migration is idempotent, so running
it multiple times and on already-migrated plugins is safe. After the migration
is completed, you should <a
href="#connect_plugindiscovery_compatibility">verify the migration is
complete</a>. These migration and verification steps may be put into a
Continuous Integration process to automatically migrate plugins.</p>
+
+ <h5><a id="connect_plugindiscovery_migratesource"
href="#connect_plugindiscovery_migratesource">Developers: Source
Migration</a></h5>
+
+ <p>To make plugins compatible with <code>SERVICE_LOAD</code>, it is
necessary to add <a
href="https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html";>ServiceLoader</a>
manifests to your source code, which should then be packaged in the release
artifact. Manifests are resource files in <code>META-INF/services/</code> named
after their superclass type, and contain a list of fully-qualified subclass
names, one on each line. Manifests should be located in the same module as the
classes they refer to, not in the corresponding test resources.</p>
+
+ <p>In order for a plugin to be compatible, it must appear as a line in a
manifest corresponding to the plugin superclass it extends. If a single plugin
implements multiple plugin interfaces, then it should appear in each
corresponding manifest. If you have no classes for a certain type of plugin,
you do not need to include a manifest file for that type. If you have classes
which should not be visible as plugins, they should be marked abstract. The
following types are expected to have manifests:</p>
+
+ <ul>
+ <li><code>org.apache.kafka.connect.sink.SinkConnector</code></li>
+ <li><code>org.apache.kafka.connect.source.SourceConnector</code></li>
+ <li><code>org.apache.kafka.connect.storage.Converter</code></li>
+ <li><code>org.apache.kafka.connect.storage.HeaderConverter</code></li>
+
<li><code>org.apache.kafka.connect.transforms.Transformation</code></li>
+
<li><code>org.apache.kafka.connect.transforms.predicates.Predicate</code></li>
+
<li><code>org.apache.kafka.common.config.provider.ConfigProvider</code></li>
+
<li><code>org.apache.kafka.connect.rest.ConnectRestExtension</code></li>
+
<li><code>org.apache.kafka.connect.connector.policy.ConnectorClientConfigOverridePolicy</code></li>
+ </ul>
+
+ <p>For example, if you only have one connector with the fully-qualified
name <code>com.example.MySinkConnector</code>, then only one manifest file must
be added to resources in
<code>META-INF/services/org.apache.kafka.connect.sink.SinkConnector</code>, and
the contents should be similar to the following:</p>
+
+ <pre class="brush: resource;">
+# license header or comment
+com.example.MySinkConnector</pre>
+
+ <p>You should then verify that your manifests are correct. either with the
<a href="#connect_plugindiscovery_compatibility">manual verification steps</a>
or with unit tests. If you aren't using the <code>EmbeddedConnectCluster</code>
you can add a test suite that starts up an <code>EmbeddedConnectCluster</code>
and does nothing with it. If you are using the
<code>EmbeddedConnectCluster</code>, then ensure you are using version 3.6 or
later of the connect-runtime main and test artifacts. If you have a test that
passes using the default <code>HYBRID_FAIL</code> strategy, then your migration
is complete. You can then release the plugin normally, and operators can
upgrade to the compatible version.</p>
Review Comment:
Also, same thought RE explicit mention of the `EmbeddedConnectCluster`
class. The instructions on starting a worker in a testing environment with
`HYBRID_FAIL` mode are great, we should just be a tad more general about how we
describe that kind of setup. This is also useful since some users have their
own integration testing environments which don't use the
`EmbeddedConnectCluster` class but can still use the `HYBRID_FAIL` testing
strategy.
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
+
+ <ul>
+ <li>Start your worker with the default
<code>HYBRID_WARN</code>strategy, and WARN logs enabled for the
<code>org.apache.kafka.connect</code> package. At least one WARN log message
mentioning the <code>plugin.discovery</code> configuration should be printed.
This log message will explicitly say that all plugins are compatible, or list
the incompatible plugins.</li>
+ <li>Start your worker in a test environment with
<code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed.
If at least one plugin is not compatible the worker will fail to start up, and
all incompatible plugins will be listed in the exception.</li>
+ </ul>
+
+ <p>If the verification step succeeds, then your current set of installed
plugins are compatible, and it should be safe to change the
<code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If
you change the set of already-installed plugins, they may no longer be
compatible, and you should repeat the above verification. If the verification
fails, you must address the incompatible plugins before using the
<code>SERVICE_LOAD</code> strategy.</p>
+
+ <h5><a id="connect_plugindiscovery_migrateartifact"
href="#connect_plugindiscovery_migrateartifact">Operators: Artifact
Migration</a></h5>
+
+ <p>As an operator of Connect, if you discover incompatible plugins, there
are multiple ways to try to resolve the incompatibility. They are listed below
from most to least preferable.</p>
+
+ <ol>
+ <li>Upgrade your incompatible plugins to the latest release version
from your plugin provider.</li>
+ <li>Contact your plugin provider and request that they migrate the
plugin to be compatible, following the <a
href="#connect_plugindiscovery_migratesource">source migration
instructions</a>, and then upgrade to the migrated version.</li>
+ <li>Migrate the plugin artifacts yourself using the included migration
script.</li>
+ </ol>
+
+ <p>The migration script is located in
<code>bin/connect-plugin-path.sh</code> and
<code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation.
The script can migrate incompatible plugin artifacts already installed on your
Connect worker's <code>plugin.path</code> by adding or modifying JAR or
resource files. This is not suitable for environments using code-signing, as
this may change the artifacts such that they will fail signature verification.
View the built-in help with <code>--help</code>.</p>
+
+ <p>To perform a migration, first use the <code>list</code> subcommand to
get an overview of the plugins available to the script. You must tell the
script where to find plugins, which can be done with the repeatable
<code>--worker-config</code>, <code>--plugin-path</code>, and
<code>--plugin-location</code> arguments. The script will only migrate plugins
present in the paths specified, so if you add plugins to your worker's
classpath, then you will need to specify those plugins via one or more
<code>--plugin-location</code> arguments.</p>
+
+ <p>Once you see that all incompatible plugins are included in the listing,
you can proceed to dry-run the migration with <code>sync-manifests
--dry-run</code>. This will perform all parts of the migration, except for
writing the results of the migration to disk. Note that the
<code>sync-manifests</code> command requires all specified paths to be
writable, and may alter the contents of the directories. Make a backup of the
specified paths, or copy them to a writable directory.</p>
+
+ <p>Ensure that you have a backup and the dry-run succeeds before removing
the <code>--dry-run</code> flag and actually running the migration. If the
migration fails without the <code>--dry-run</code> flag, then the partially
migrated artifacts should be discarded. The migration is idempotent, so running
it multiple times and on already-migrated plugins is safe. After the migration
is completed, you should <a
href="#connect_plugindiscovery_compatibility">verify the migration is
complete</a>. These migration and verification steps may be put into a
Continuous Integration process to automatically migrate plugins.</p>
+
+ <h5><a id="connect_plugindiscovery_migratesource"
href="#connect_plugindiscovery_migratesource">Developers: Source
Migration</a></h5>
+
+ <p>To make plugins compatible with <code>SERVICE_LOAD</code>, it is
necessary to add <a
href="https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html";>ServiceLoader</a>
manifests to your source code, which should then be packaged in the release
artifact. Manifests are resource files in <code>META-INF/services/</code> named
after their superclass type, and contain a list of fully-qualified subclass
names, one on each line. Manifests should be located in the same module as the
classes they refer to, not in the corresponding test resources.</p>
+
+ <p>In order for a plugin to be compatible, it must appear as a line in a
manifest corresponding to the plugin superclass it extends. If a single plugin
implements multiple plugin interfaces, then it should appear in each
corresponding manifest. If you have no classes for a certain type of plugin,
you do not need to include a manifest file for that type. If you have classes
which should not be visible as plugins, they should be marked abstract. The
following types are expected to have manifests:</p>
Review Comment:
Nit: maybe instead of "If a single plugin implements multiple plugin
interfaces, then it should appear in each corresponding manifest" we can say
"If a single plugin implements multiple plugin interfaces, then it should
appear in a manifest for each plugin interface that it implements"?
##########
docs/connect.html:
##########
@@ -583,6 +644,10 @@ <h5><a id="connect_connectorexample"
href="#connect_connectorexample">Connector
public class FileStreamSourceConnector extends SourceConnector {
private Map<String, String> props;</pre>
+ <p>Add a corresponding ServiceLoader manifest to your resources in
<code>META-INF/services/org.apache.kafka.connect.source.SourceConnector</code>
to allow it to be discovered by the runtime.</p>
Review Comment:
This kind of breaks the flow of writing our connector class; it comes
between the `public class FileStreamConnector...` declaration and the `public
Class<? extends Task> taskClass()` declaration.
Maybe we can put this before the class declaration, with a note that the
name of the connector we're developing will be `FileStreamSourceConnector`?
Also, we should probably link to the docs on plugin discovery modes in this
section. Maybe something like "See <a href="#connect_plugindiscovery">plugin
discovery</a> for more info on this step"?
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
+
+ <ul>
+ <li>Start your worker with the default
<code>HYBRID_WARN</code>strategy, and WARN logs enabled for the
<code>org.apache.kafka.connect</code> package. At least one WARN log message
mentioning the <code>plugin.discovery</code> configuration should be printed.
This log message will explicitly say that all plugins are compatible, or list
the incompatible plugins.</li>
+ <li>Start your worker in a test environment with
<code>HYBRID_FAIL</code>. If all plugins are compatible, startup will succeed.
If at least one plugin is not compatible the worker will fail to start up, and
all incompatible plugins will be listed in the exception.</li>
+ </ul>
+
+ <p>If the verification step succeeds, then your current set of installed
plugins are compatible, and it should be safe to change the
<code>plugin.discovery</code> configuration to <code>SERVICE_LOAD</code>. If
you change the set of already-installed plugins, they may no longer be
compatible, and you should repeat the above verification. If the verification
fails, you must address the incompatible plugins before using the
<code>SERVICE_LOAD</code> strategy.</p>
+
+ <h5><a id="connect_plugindiscovery_migrateartifact"
href="#connect_plugindiscovery_migrateartifact">Operators: Artifact
Migration</a></h5>
+
+ <p>As an operator of Connect, if you discover incompatible plugins, there
are multiple ways to try to resolve the incompatibility. They are listed below
from most to least preferable.</p>
+
+ <ol>
+ <li>Upgrade your incompatible plugins to the latest release version
from your plugin provider.</li>
+ <li>Contact your plugin provider and request that they migrate the
plugin to be compatible, following the <a
href="#connect_plugindiscovery_migratesource">source migration
instructions</a>, and then upgrade to the migrated version.</li>
+ <li>Migrate the plugin artifacts yourself using the included migration
script.</li>
+ </ol>
+
+ <p>The migration script is located in
<code>bin/connect-plugin-path.sh</code> and
<code>bin\windows\connect-plugin-path.bat</code> of your Kafka installation.
The script can migrate incompatible plugin artifacts already installed on your
Connect worker's <code>plugin.path</code> by adding or modifying JAR or
resource files. This is not suitable for environments using code-signing, as
this may change the artifacts such that they will fail signature verification.
View the built-in help with <code>--help</code>.</p>
+
+ <p>To perform a migration, first use the <code>list</code> subcommand to
get an overview of the plugins available to the script. You must tell the
script where to find plugins, which can be done with the repeatable
<code>--worker-config</code>, <code>--plugin-path</code>, and
<code>--plugin-location</code> arguments. The script will only migrate plugins
present in the paths specified, so if you add plugins to your worker's
classpath, then you will need to specify those plugins via one or more
<code>--plugin-location</code> arguments.</p>
+
+ <p>Once you see that all incompatible plugins are included in the listing,
you can proceed to dry-run the migration with <code>sync-manifests
--dry-run</code>. This will perform all parts of the migration, except for
writing the results of the migration to disk. Note that the
<code>sync-manifests</code> command requires all specified paths to be
writable, and may alter the contents of the directories. Make a backup of the
specified paths, or copy them to a writable directory.</p>
+
+ <p>Ensure that you have a backup and the dry-run succeeds before removing
the <code>--dry-run</code> flag and actually running the migration. If the
migration fails without the <code>--dry-run</code> flag, then the partially
migrated artifacts should be discarded. The migration is idempotent, so running
it multiple times and on already-migrated plugins is safe. After the migration
is completed, you should <a
href="#connect_plugindiscovery_compatibility">verify the migration is
complete</a>. These migration and verification steps may be put into a
Continuous Integration process to automatically migrate plugins.</p>
+
+ <h5><a id="connect_plugindiscovery_migratesource"
href="#connect_plugindiscovery_migratesource">Developers: Source
Migration</a></h5>
+
+ <p>To make plugins compatible with <code>SERVICE_LOAD</code>, it is
necessary to add <a
href="https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html";>ServiceLoader</a>
manifests to your source code, which should then be packaged in the release
artifact. Manifests are resource files in <code>META-INF/services/</code> named
after their superclass type, and contain a list of fully-qualified subclass
names, one on each line. Manifests should be located in the same module as the
classes they refer to, not in the corresponding test resources.</p>
Review Comment:
Is the "Manifests should be located in the same module as the classes they
refer to, not in the corresponding test resources." part necessary? When do we
expect users to do something like this?
##########
docs/connect.html:
##########
@@ -543,6 +543,67 @@ <h6>ACL requirements</h6>
</tbody>
</table>
+ <h4><a id="connect_plugindiscovery" href="#connect_plugindiscovery">Plugin
Discovery</a></h4>
+
+ <p>Plugin discovery is the name for the strategy which the Connect worker
uses to find plugin classes and make them accessible to configure and run in
Connectors and Tasks. This is controlled by the <a
href="#connectconfigs_plugin.discovery"><code>plugin.discovery</code> worker
configuration</a>, and has a significant impact on worker startup time.
<code>SERVICE_LOAD</code> is the fastest strategy, but care should be taken to
verify that plugins are compatible before setting this configuration to
<code>SERVICE_LOAD</code>.</p>
+
+ <p>Prior to version 3.6, this strategy was not configurable, and behaved
like the <code>ONLY_SCAN</code> mode which is compatible with all plugins. For
version 3.6 and later, this mode defaults to <code>HYBRID_WARN</code> which is
also compatible with all plugins, but logs a warning for all plugins which are
incompatible with the other modes. For unit-test environments that use the
<code>EmbeddedConnectCluster</code> this defaults to the
<code>HYBRID_FAIL</code> strategy, which stops the worker with an error if an
incompatible plugin is detected. Finally, the <code>SERVICE_LOAD</code>
strategy will silently hide incompatible plugins and make them unusable.</p>
+
+ <h5><a id="connect_plugindiscovery_compatibility"
href="#connect_plugindiscovery_compatibility">Verifying Plugin
Compatibility</a></h5>
+
+ <p>To verify if all of your plugins are compatible, first ensure that you
are using version 3.6 or later of the Connect runtime. You can then perform one
of the following checks:</p>
Review Comment:
Can just say "Kafka Connect" instead of "the Connect runtime".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
