[
https://issues.apache.org/jira/browse/SOLR-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul updated SOLR-13661:
------------------------------
Description:
Solr needs a unified cohesive package management system so that users can
deploy/redeploy plugins in a safe manner. This is an umbrella issue to
eventually build that solution
h2. What is a package?
A package has 2 parts
* The {{jar}} binaries which may contain standard plugins (or even other
libraries). It does not expect anything special. Basically , any jar file can
be added into a package
* The configuration in ZK. The structure is as follows
{code:json}
{
"name": "package-name",
"version": "some-version-string",
"blob": {
"sha256": "the-sha256-of-the-blob",
"sig": "signature-signed-with-your-private-key",
"name": "some-identifier"
}
}
// use the following format for your multi jar package
{
"name": "package-name",
"version": "some-version-string",
"blobs": [
{
"sha256": "the-sha256-of-the-blob",
"sig": "signature-signed-with-your-private-key",
"name": "some-identifier"
}
]
}
{code}
* {{*name*}} : this is identifier used to access the package
* {{*version*}}: This is not consumed by the system. But, it is required so
that the user can make out which version is being used
* {{*blob/sha256*}}: The sha256 of the blob. You should have uploaded this
blob into the new FS blob store
* {{*blob/sig*}} : signature. ensure that your public keys are already there
in {{/keys/exe}} in ZK
* {{blob/name}} : a friendly identifier (not required)
Voila! this is all you need to have a package in Solr
Every package gets its own isolated unique classloader in a Solr node. The
classloader is a child of the {{CoreContainer}} classloader. Basically, your
classes have to be either in the root classloader or in the blobs in your
package
h2. PackageLoader Components
h3. {{PackageResourceLoader}} class
This is a standard {{SolrResourceLoader}} . It has all the blobs of a package
in its classpath.
h3. {{PackageManager}} class
This is a registry of all Package classloaders . essentially a {{Map<String,
PackageResourceLoader>}}
This class is responsible for listening to the state changes in ZK to the
packages ADD/UPDATE/DELETE operations.
* ADD : create a new {{PackageResourceLoader}} and add it to the registry
* UPDATE: create a new {{PackageResourceLoader}}, update the registry and
notify every listener
* DELETE: remove a {{PackageResourceLoader}}
h4. Public APIs
* {{GET /api/cluster/packages}} : Get a list of all registered packages
* {{POST /api/cluster/packages}}
** {{add}} command: create a new package
** {{add}} command: update an existing package
** {{delete}} command: delete an existing package
h4. Configuring your plugins to use packages
h5. in your {{solrconfig.xml}}
{code:xml}
<requestHandler name="/my_handler" class="my.full.ClassName" package="pkg_name">
</requestHandler>
{code}
h5. Using config API to use a package
{code:xml}
{
"add-requesthandler": {"name":""/my_handler", "class":"my.full.ClassName",
"package":"pkg_name" }}
{code}
h5. Hot loading schema from package (TBD)
{code:java}
<schemaFactory class="ClassicIndexSchemaFactory" package= "pkg_name"/>
{code}
h4. Verifying your components
{code:java}
GET /solr/<collection-name>/config/<component-type>/component-name?meta=true
{code}
This is the standard config API. with an extra {{meta=true}} parameter
This API would return the component configuration and curerent package being
used as follows
{code:json}
{
"config": {
"requestHandler": {
"/my_handler": {
"name": "/my_handler",
"class": "org.apache.solr.core.RuntimeLibReqHandler",
"package": "global",
"_packageinfo_": {
"name": "global",
"version": "1",
"blob": {
"sha256":
"e1f9e23988c19619402f1040c9251556dcd6e02b9d3e3b966a129ea1be5c70fc",
"sig":
"L3q/qIGs4NaF6JiO0ZkMUFa88j0OmYc+I6O7BOdNuMct/xoZ4h73aZHZGc0+nmI1f/U3bOlMPINlSOM6LK3JpQ=="
}
}
}
}}}
{code}
h3. The Listeners
Listeners listen to changes to packages. They get notified and they can take
appropriate action
h3. {{PackagePluginHolder}}
This is a special {{PluginHolder}} that impements {{PackageListener}} .
Whenever it gets a notification of a package update , it reinitializes the
plugin it's holding and throw away the old one
h2. {{FsBlobStore}} a.k.a FIle System Blob Store
Why a new blob store? why file system?
* File systems are reliable and always available. (It's available even before
the first line of code is executed)
** Which means we can load every single Solr component from it. No dependency
on some collection to be available or loaded
* Users can use it to cold load blobs in standard docker (and other) images
* Users can directly add remove blobs from file system, if required. They
already have the tools to do so
( Though we don't recommend it)
This is a fully replicated file system blob store.
* It lives at {{<solr.home>/blobs}} on each Solr node
* Every blob is a file. The file name is the sha256 of the blob
* It exposes itself at the {{/api/node/blob}} end point in each node
When the {{FsBlobStore}} is requested a blob, it
* looks up in the local file store for the file
* If not found, it asks the blob store in every other node in the cluster one
by one
** After it finds a node, the blob is downloaded and persisted to the local
file system
h3. Can it be used for non-jar blobs?
Yes, the {{FsBlobStore}} is totally agnostic of the contents of the blob. You
may store other files in it if required.
h3. Public APIs
* {{POST /api/cluster/blob}} : create a new blob in the system. The {{sha256}}
is computed and it is saved as a file.
* * This API asks other nodes to download the blob.
* {{GET /api/node/blob}} : List of all the locally available blobs
* {{GET /api/node/blob/<sha256>}} : Download a blob
* {{DELETE /api/cluster/bolob/<sha256>}} delete a blob from all nodes (TBD)
h2. Security
* The package loader is disabled by default. Start your nodes with
{{-Denable.package=true}}. This is to protect users who upgrade to solr 8.3
without taking measures to protect their cluster
* Signing jars is a must. You must create . your key pair and upload the
public keys to ZK for this to work.
* Ensure that nobody can write to your ZK
* Ensure that security is enabled and access to blob store/ package API is
given only to admins
h2. The CLI PackageManager
<TBD>
was:
Solr needs a unified cohesive package management system so that users can
deploy/redeploy plugins in a safe manner. This is an umbrella issue to
eventually build that solution
h2. What is a package?
A package has 2 parts
* The {{jar}} binaries which may contain standard plugins (or even other
libraries). It does not expect anything special. Basically , any jar file can
be added into a package
* The configuration in ZK. The structure is as follows
{code:json}
{
"name": "package-name",
"version": "some-version-string",
"blob": {
"sha256": "the-sha256-of-the-blob",
"sig": "signature-signed-with-your-private-key",
"name": "some-identifier"
}
}
// use the following format for your multi jar package
{
"name": "package-name",
"version": "some-version-string",
"blobs": [
{
"sha256": "the-sha256-of-the-blob",
"sig": "signature-signed-with-your-private-key",
"name": "some-identifier"
}
]
}
{code}
* {{*name*}} : this is identifier used to access the package
* {{*version*}}: This is not consumed by the system. But, it is required so
that the user can make out which version is being used
* {{*blob/sha256*}}: The sha256 of the blob. You should have uploaded this
blob into the new FS blob store
* {{*blob/sig*}} : signature. ensure that your public keys are already there
in {{/keys/exe}} in ZK
* {{blob/name}} : a friendly identifier (not required)
Voila! this is all you need to have a package in Solr
Every package gets its own isolated unique classloader in a Solr node. The
classloader is a child of the {{CoreContainer}} classloader. Basically, your
classes have to be either in the root classloader or in the blobs in your
package
h2. PackageLoader Components
h3. {{PackageResourceLoader}} class
This is a standard {{SolrResourceLoader}} . It has all the blobs of a package
in its classpath.
h3. {{PackageManager}} class
This is a registry of all Package classloaders . essentially a {{Map<String,
PackageResourceLoader>}}
This class is responsible for listening to the state changes in ZK to the
packages ADD/UPDATE/DELETE operations.
* ADD : create a new {{PackageResourceLoader}} and add it to the registry
* UPDATE: create a new {{PackageResourceLoader}}, update the registry and
notify every listener
* DELETE: remove a {{PackageResourceLoader}}
h4. Public APIs
* {{GET /api/cluster/packages}} : Get a list of all registered packages
* {{POST /api/cluster/packages}}
** {{add}} command: create a new package
** {{add}} command: update an existing package
** {{delete}} command: delete an existing package
h4. Configuring your plugins to use packages
h5. in your {{solrconfig.xml}}
{code:xml}
<requestHandler name="/my_handler" class="my.full.ClassName" package="pkg_name">
</requestHandler>
{code}
h5. Using config API to use a package
{code:xml}
{
"add-requesthandler": {"name":""/my_handler", "class":"my.full.ClassName",
"package":"pkg_name" }}
{code}
h5. Hot loading schema from package (TBD)
{code:java}
<schemaFactory class="ClassicIndexSchemaFactory" package= "pkg_name"/>
{code}
h4. Verifying your components
{code:java}
GET /solr/<collection-name>/config/<component-type>/component-name?meta=true
{code}
This is the standard config API. with an extra {{meta=true}} parameter
This API would return the component configuration and curerent package being
used as follows
{code:json}
{
"config": {
"requestHandler": {
"/my_handler": {
"name": "/my_handler",
"class": "org.apache.solr.core.RuntimeLibReqHandler",
"package": "global",
"_packageinfo_": {
"name": "global",
"version": "1",
"blob": {
"sha256":
"e1f9e23988c19619402f1040c9251556dcd6e02b9d3e3b966a129ea1be5c70fc",
"sig":
"L3q/qIGs4NaF6JiO0ZkMUFa88j0OmYc+I6O7BOdNuMct/xoZ4h73aZHZGc0+nmI1f/U3bOlMPINlSOM6LK3JpQ=="
}
}
}
}}}
{code}
h3. The Listeners
Listeners listen to changes to packages. They get notified and they can take
appropriate action
h3. {{PackagePluginHolder}}
This is a special {{PluginHolder}} that impements {{PackageListener}} .
Whenever it gets a notification of a package update , it reinitializes the
plugin it's holding and throw away the old one
h2. {{FsBlobStore}} a.k.a FIle System Blob Store
Why a new blob store? why file system?
* File systems are reliable and always available. (It's available even before
the first line of code is executed)
** Which means we can load every single Solr component from it. No dependency
on some collection to be available or loaded
* Users can use it to cold load blobs in standard docker (and other) images
* Users can directly add remove blobs from file system, if required. They
already have the tools to do so
( Though we don't recommend it)
This is a fully replicated file system blob store.
* It lives at {{<solr.home>/blobs}} on each Solr node
* Every blob is a file. The file name is the sha256 of the blob
* It exposes itself at the {{/api/node/blob}} end point in each node
When the {{FsBlobStore}} is requested a blob, it
* looks up in the local file store for the file
* If not found, it asks the blob store in every other node in the cluster one
by one
** After it finds a node, the blob is downloaded and persisted to the local
file system
h3. Public APIs
* {{POST /api/cluster/blob}} : create a new blob in the system. The {{sha256}}
is computed and it is saved as a file.
* * This API asks other nodes to download the blob.
* {{GET /api/node/blob}} : List of all the locally available blobs
* {{GET /api/node/blob/<sha256>}} : Download a blob
* {{DELETE /api/cluster/bolob/<sha256>}} delete a blob from all nodes (TBD)
h2. Security
* The package loader is disabled by default. Start your nodes with
{{-Denable.package=true}}. This is to protect users who upgrade to solr 8.3
without taking measures to protect their cluster
* Signing jars is a must. You must create . your key pair and upload the
public keys to ZK for this to work.
* Ensure that nobody can write to your ZK
* Ensure that security is enabled and access to blob store/ package API is
given only to admins
h2. The CLI PackageManager
<TBD>
> A package management system for Solr
> ------------------------------------
>
> Key: SOLR-13661
> URL: https://issues.apache.org/jira/browse/SOLR-13661
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Noble Paul
> Priority: Major
> Labels: package
> Attachments: plugin-usage.png, repos.png
>
>
> Solr needs a unified cohesive package management system so that users can
> deploy/redeploy plugins in a safe manner. This is an umbrella issue to
> eventually build that solution
> h2. What is a package?
> A package has 2 parts
> * The {{jar}} binaries which may contain standard plugins (or even other
> libraries). It does not expect anything special. Basically , any jar file can
> be added into a package
> * The configuration in ZK. The structure is as follows
> {code:json}
> {
> "name": "package-name",
> "version": "some-version-string",
> "blob": {
> "sha256": "the-sha256-of-the-blob",
> "sig": "signature-signed-with-your-private-key",
> "name": "some-identifier"
> }
> }
> // use the following format for your multi jar package
> {
> "name": "package-name",
> "version": "some-version-string",
> "blobs": [
> {
> "sha256": "the-sha256-of-the-blob",
> "sig": "signature-signed-with-your-private-key",
> "name": "some-identifier"
> }
> ]
> }
> {code}
> * {{*name*}} : this is identifier used to access the package
> * {{*version*}}: This is not consumed by the system. But, it is required so
> that the user can make out which version is being used
> * {{*blob/sha256*}}: The sha256 of the blob. You should have uploaded this
> blob into the new FS blob store
> * {{*blob/sig*}} : signature. ensure that your public keys are already there
> in {{/keys/exe}} in ZK
> * {{blob/name}} : a friendly identifier (not required)
> Voila! this is all you need to have a package in Solr
> Every package gets its own isolated unique classloader in a Solr node. The
> classloader is a child of the {{CoreContainer}} classloader. Basically, your
> classes have to be either in the root classloader or in the blobs in your
> package
> h2. PackageLoader Components
> h3. {{PackageResourceLoader}} class
> This is a standard {{SolrResourceLoader}} . It has all the blobs of a package
> in its classpath.
> h3. {{PackageManager}} class
> This is a registry of all Package classloaders . essentially a {{Map<String,
> PackageResourceLoader>}}
> This class is responsible for listening to the state changes in ZK to the
> packages ADD/UPDATE/DELETE operations.
> * ADD : create a new {{PackageResourceLoader}} and add it to the registry
> * UPDATE: create a new {{PackageResourceLoader}}, update the registry and
> notify every listener
> * DELETE: remove a {{PackageResourceLoader}}
> h4. Public APIs
> * {{GET /api/cluster/packages}} : Get a list of all registered packages
> * {{POST /api/cluster/packages}}
> ** {{add}} command: create a new package
> ** {{add}} command: update an existing package
> ** {{delete}} command: delete an existing package
> h4. Configuring your plugins to use packages
> h5. in your {{solrconfig.xml}}
> {code:xml}
> <requestHandler name="/my_handler" class="my.full.ClassName"
> package="pkg_name">
> </requestHandler>
> {code}
> h5. Using config API to use a package
> {code:xml}
> {
> "add-requesthandler": {"name":""/my_handler", "class":"my.full.ClassName",
> "package":"pkg_name" }}
> {code}
> h5. Hot loading schema from package (TBD)
> {code:java}
> <schemaFactory class="ClassicIndexSchemaFactory" package= "pkg_name"/>
> {code}
> h4. Verifying your components
> {code:java}
> GET /solr/<collection-name>/config/<component-type>/component-name?meta=true
> {code}
> This is the standard config API. with an extra {{meta=true}} parameter
> This API would return the component configuration and curerent package being
> used as follows
> {code:json}
> {
> "config": {
> "requestHandler": {
> "/my_handler": {
> "name": "/my_handler",
> "class": "org.apache.solr.core.RuntimeLibReqHandler",
> "package": "global",
> "_packageinfo_": {
> "name": "global",
> "version": "1",
> "blob": {
> "sha256":
> "e1f9e23988c19619402f1040c9251556dcd6e02b9d3e3b966a129ea1be5c70fc",
> "sig":
> "L3q/qIGs4NaF6JiO0ZkMUFa88j0OmYc+I6O7BOdNuMct/xoZ4h73aZHZGc0+nmI1f/U3bOlMPINlSOM6LK3JpQ=="
> }
> }
> }
> }}}
> {code}
> h3. The Listeners
> Listeners listen to changes to packages. They get notified and they can take
> appropriate action
> h3. {{PackagePluginHolder}}
> This is a special {{PluginHolder}} that impements {{PackageListener}} .
> Whenever it gets a notification of a package update , it reinitializes the
> plugin it's holding and throw away the old one
> h2. {{FsBlobStore}} a.k.a FIle System Blob Store
> Why a new blob store? why file system?
> * File systems are reliable and always available. (It's available even
> before the first line of code is executed)
> ** Which means we can load every single Solr component from it. No
> dependency on some collection to be available or loaded
> * Users can use it to cold load blobs in standard docker (and other) images
> * Users can directly add remove blobs from file system, if required. They
> already have the tools to do so
> ( Though we don't recommend it)
> This is a fully replicated file system blob store.
> * It lives at {{<solr.home>/blobs}} on each Solr node
> * Every blob is a file. The file name is the sha256 of the blob
> * It exposes itself at the {{/api/node/blob}} end point in each node
> When the {{FsBlobStore}} is requested a blob, it
> * looks up in the local file store for the file
> * If not found, it asks the blob store in every other node in the cluster
> one by one
> ** After it finds a node, the blob is downloaded and persisted to the local
> file system
> h3. Can it be used for non-jar blobs?
> Yes, the {{FsBlobStore}} is totally agnostic of the contents of the blob. You
> may store other files in it if required.
> h3. Public APIs
> * {{POST /api/cluster/blob}} : create a new blob in the system. The
> {{sha256}} is computed and it is saved as a file.
> * * This API asks other nodes to download the blob.
> * {{GET /api/node/blob}} : List of all the locally available blobs
> * {{GET /api/node/blob/<sha256>}} : Download a blob
> * {{DELETE /api/cluster/bolob/<sha256>}} delete a blob from all nodes (TBD)
> h2. Security
> * The package loader is disabled by default. Start your nodes with
> {{-Denable.package=true}}. This is to protect users who upgrade to solr 8.3
> without taking measures to protect their cluster
> * Signing jars is a must. You must create . your key pair and upload the
> public keys to ZK for this to work.
> * Ensure that nobody can write to your ZK
> * Ensure that security is enabled and access to blob store/ package API is
> given only to admins
> h2. The CLI PackageManager
> <TBD>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
