Re: Solr Cloud on Docker?

[Jan Høydahl]

We have been experimenting with Solr cloud in Docker for some while,
and tried to do some optimizations by turning off swap on the host completely.
However, that quickly led to OOM crashes although we had 8G physical, 4G heap
and Solr just having a few thousand docs.

This makes me suspect that the common advice of disabling swap may not work
that well when Docker is involved, due to how Docker manages memory. I found 
some
evidence that Docker often will use swap first even if physical memory is 
available.

We ended up enabling swap again, but instead reduce vm.swappiness to 1.

So any information or best practices regarding Swap/swappiness and perhaps also
the docker options --memory-swap, --memory-swappiness.

Any experience from you guys would be valuable!

Jan

> 5. feb. 2020 kl. 20:01 skrev Karl Stoney 
> :
> 
> Nothing much to add to the below apart from we also successfully run solr on 
> kubernetes.  It took some implementation effort but we're now at a point 
> where we can do `kubectl scale --replicas=x statefulset/solr` and increase 
> capacity in minutes with solr's autoscaling taking care of the new shard 
> creation.
> 
> Very happy.
> 
> From: Dominique Bejean  <mailto:dominique.bej...@eolya.fr>>
> Sent: 05 February 2020 17:53
> To: Dwane Hall mailto:dwaneh...@hotmail.com>>
> Cc: Scott Stults  <mailto:sstu...@opensourceconnections.com>>; solr-user@lucene.apache.org 
> <mailto:solr-user@lucene.apache.org>  <mailto:solr-user@lucene.apache.org>>
> Subject: Re: Solr Cloud on Docker?
> 
> Thank you Dwane. Great info :)
> 
> 
> Le mer. 5 févr. 2020 à 11:49, Dwane Hall  a écrit :
> 
>> Hey Dominique,
>> 
>> From a memory management perspective I don't do any container resource
>> limiting specifically in Docker (although as you mention you certainly
>> can).  In our circumstances these hosts are used specifically for Solr so I
>> planned and tested my capacity beforehand. We have ~768G of RAM on each of
>> these 5 hosts so with 20x16G heaps we had ~320G of heap being used by Solr,
>> some overhead for Docker and the other OS services leaving ~400G for the OS
>> cache and whatever wants to grab it on each host. Not everyone will have
>> servers this large which is why we really had to take advantage of multiple
>> Solr instances/host and Docker became important for our cluster operation
>> management.  Our disk's are not SSD's either and all instances write to the
>> same raid 5 spinner which is bind mounted to the containers.  With this
>> configuration we've been able to achieve consistent median response times
>> of under 500ms across the largest collection but obviously query type
>> varies this (no terms, leading wildcards etc.).  Our QPS is not huge
>> ranging from 2-20/sec but if we need to scale further or speed up response
>> times there's certainly wins that can be made at a disk level.  For our
>> current circumstances we're very content with the deployment.
>> 
>> In not sure if you've read Toke's blog on his experiences at the Royal
>> Danish Library but I found it really useful when capacity planning and
>> recommend reading it (
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsbdevel.wordpress.com%2F2016%2F11%2F30%2F70tb-16b-docs-4-machines-1-solrcloud%2Fdata=02%7C01%7Ckarl.stoney%40autotrader.co.uk%7C551dd53ab648462d6ae008d7aa6463d4%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637165220483158911sdata=LXYgh3kUAo4X4mDbDIqhJO%2B%2BR3FdrTxci3sNw%2Frm0sc%3Dreserved=0
>>  
>> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsbdevel.wordpress.com%2F2016%2F11%2F30%2F70tb-16b-docs-4-machines-1-solrcloud%2Fdata=02%7C01%7Ckarl.stoney%40autotrader.co.uk%7C551dd53ab648462d6ae008d7aa6463d4%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637165220483158911sdata=LXYgh3kUAo4X4mDbDIqhJO%2B%2BR3FdrTxci3sNw%2Frm0sc%3Dreserved=0>
>> ).
>> 
>> As always it's recommend to test for your own conditions and best of luck
>> with your deployment!
>> 
>> Dwane
>> 
>> --
>> *From:* Scott Stults > <mailto:sstu...@opensourceconnections.com>>
>> *Sent:* Thursday, 30 January 2020 1:45 AM
>> *To:* solr-user@lucene.apache.org <mailto:solr-user@lucene.apache.org> 
>> mailto:solr-user@lucene.apache.org>>
>> *Subject:* Re: Solr Cloud on Docker?
>> 
>> One of our clients has been running a big Solr Cloud (100-ish nodes, TB
>> index, billions of docs) in kubernetes for over a year and it's been
>> wonderful. I think during that time the biggest scrapes we got were when we
>> ran out of disk space. Performance and reliab

Re: Solr Cloud on Docker?

[Karl Stoney]

Nothing much to add to the below apart from we also successfully run solr on 
kubernetes.  It took some implementation effort but we're now at a point where 
we can do `kubectl scale --replicas=x statefulset/solr` and increase capacity 
in minutes with solr's autoscaling taking care of the new shard creation.

Very happy.

From: Dominique Bejean 
Sent: 05 February 2020 17:53
To: Dwane Hall 
Cc: Scott Stults ; 
solr-user@lucene.apache.org 
Subject: Re: Solr Cloud on Docker?

Thank you Dwane. Great info :)


Le mer. 5 févr. 2020 à 11:49, Dwane Hall  a écrit :

> Hey Dominique,
>
> From a memory management perspective I don't do any container resource
> limiting specifically in Docker (although as you mention you certainly
> can).  In our circumstances these hosts are used specifically for Solr so I
> planned and tested my capacity beforehand. We have ~768G of RAM on each of
> these 5 hosts so with 20x16G heaps we had ~320G of heap being used by Solr,
> some overhead for Docker and the other OS services leaving ~400G for the OS
> cache and whatever wants to grab it on each host. Not everyone will have
> servers this large which is why we really had to take advantage of multiple
> Solr instances/host and Docker became important for our cluster operation
> management.  Our disk's are not SSD's either and all instances write to the
> same raid 5 spinner which is bind mounted to the containers.  With this
> configuration we've been able to achieve consistent median response times
> of under 500ms across the largest collection but obviously query type
> varies this (no terms, leading wildcards etc.).  Our QPS is not huge
> ranging from 2-20/sec but if we need to scale further or speed up response
> times there's certainly wins that can be made at a disk level.  For our
> current circumstances we're very content with the deployment.
>
> In not sure if you've read Toke's blog on his experiences at the Royal
> Danish Library but I found it really useful when capacity planning and
> recommend reading it (
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsbdevel.wordpress.com%2F2016%2F11%2F30%2F70tb-16b-docs-4-machines-1-solrcloud%2Fdata=02%7C01%7Ckarl.stoney%40autotrader.co.uk%7C551dd53ab648462d6ae008d7aa6463d4%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637165220483158911sdata=LXYgh3kUAo4X4mDbDIqhJO%2B%2BR3FdrTxci3sNw%2Frm0sc%3Dreserved=0
> ).
>
> As always it's recommend to test for your own conditions and best of luck
> with your deployment!
>
> Dwane
>
> --
> *From:* Scott Stults 
> *Sent:* Thursday, 30 January 2020 1:45 AM
> *To:* solr-user@lucene.apache.org 
> *Subject:* Re: Solr Cloud on Docker?
>
> One of our clients has been running a big Solr Cloud (100-ish nodes, TB
> index, billions of docs) in kubernetes for over a year and it's been
> wonderful. I think during that time the biggest scrapes we got were when we
> ran out of disk space. Performance and reliability has been solid
> otherwise. Like Dwane alluded to, a lot of operations pitfalls can be
> avoided if you do your Docker orchestration through kubernetes.
>
>
> k/r,
> Scott
>
> On Tue, Jan 28, 2020 at 3:34 AM Dominique Bejean <
> dominique.bej...@eolya.fr>
> wrote:
>
> > Hi  Dwane,
> >
> > Thank you for sharing this great solr/docker user story.
> >
> > According to your Solr/JVM memory requirements (Heap size + MetaSpace +
> > OffHeap size) are you specifying specific settings in docker-compose
> files
> > (mem_limit, mem_reservation, mem_swappiness, ...) ?
> > I suppose you are limiting total memory used by all dockerised Solr in
> > order to keep free memory on host for MMAPDirectory ?
> >
> > In short can you explain the memory management ?
> >
> > Regards
> >
> > Dominique
> >
> >
> >
> >
> > Le lun. 23 déc. 2019 à 00:17, Dwane Hall  a
> écrit :
> >
> > > Hey Walter,
> > >
> > > I recently migrated our Solr cluster to Docker and am very pleased I
> did
> > > so. We run relativity large servers and run multiple Solr instances per
> > > physical host and having managed Solr upgrades on bare metal installs
> > since
> > > Solr 5, containerisation has been a blessing (currently Solr 7.7.2). In
> > our
> > > case we run 20 Solr nodes per host over 5 hosts totalling 100 Solr
> > > instances. Here I host 3 collections of varying size. The first
> contains
> > > 60m docs (8 shards), the second 360m (12 shards) , and the third 1.3b
> (30
> > > shards) all with 2 NRT replicas. The docs are primarily database
> sourced
> > > but are not tiny by any means.
>

Re: Solr Cloud on Docker?

[Dominique Bejean]

Thank you Dwane. Great info :)


Le mer. 5 févr. 2020 à 11:49, Dwane Hall  a écrit :

> Hey Dominique,
>
> From a memory management perspective I don't do any container resource
> limiting specifically in Docker (although as you mention you certainly
> can).  In our circumstances these hosts are used specifically for Solr so I
> planned and tested my capacity beforehand. We have ~768G of RAM on each of
> these 5 hosts so with 20x16G heaps we had ~320G of heap being used by Solr,
> some overhead for Docker and the other OS services leaving ~400G for the OS
> cache and whatever wants to grab it on each host. Not everyone will have
> servers this large which is why we really had to take advantage of multiple
> Solr instances/host and Docker became important for our cluster operation
> management.  Our disk's are not SSD's either and all instances write to the
> same raid 5 spinner which is bind mounted to the containers.  With this
> configuration we've been able to achieve consistent median response times
> of under 500ms across the largest collection but obviously query type
> varies this (no terms, leading wildcards etc.).  Our QPS is not huge
> ranging from 2-20/sec but if we need to scale further or speed up response
> times there's certainly wins that can be made at a disk level.  For our
> current circumstances we're very content with the deployment.
>
> In not sure if you've read Toke's blog on his experiences at the Royal
> Danish Library but I found it really useful when capacity planning and
> recommend reading it (
> https://sbdevel.wordpress.com/2016/11/30/70tb-16b-docs-4-machines-1-solrcloud/
> ).
>
> As always it's recommend to test for your own conditions and best of luck
> with your deployment!
>
> Dwane
>
> --
> *From:* Scott Stults 
> *Sent:* Thursday, 30 January 2020 1:45 AM
> *To:* solr-user@lucene.apache.org 
> *Subject:* Re: Solr Cloud on Docker?
>
> One of our clients has been running a big Solr Cloud (100-ish nodes, TB
> index, billions of docs) in kubernetes for over a year and it's been
> wonderful. I think during that time the biggest scrapes we got were when we
> ran out of disk space. Performance and reliability has been solid
> otherwise. Like Dwane alluded to, a lot of operations pitfalls can be
> avoided if you do your Docker orchestration through kubernetes.
>
>
> k/r,
> Scott
>
> On Tue, Jan 28, 2020 at 3:34 AM Dominique Bejean <
> dominique.bej...@eolya.fr>
> wrote:
>
> > Hi  Dwane,
> >
> > Thank you for sharing this great solr/docker user story.
> >
> > According to your Solr/JVM memory requirements (Heap size + MetaSpace +
> > OffHeap size) are you specifying specific settings in docker-compose
> files
> > (mem_limit, mem_reservation, mem_swappiness, ...) ?
> > I suppose you are limiting total memory used by all dockerised Solr in
> > order to keep free memory on host for MMAPDirectory ?
> >
> > In short can you explain the memory management ?
> >
> > Regards
> >
> > Dominique
> >
> >
> >
> >
> > Le lun. 23 déc. 2019 à 00:17, Dwane Hall  a
> écrit :
> >
> > > Hey Walter,
> > >
> > > I recently migrated our Solr cluster to Docker and am very pleased I
> did
> > > so. We run relativity large servers and run multiple Solr instances per
> > > physical host and having managed Solr upgrades on bare metal installs
> > since
> > > Solr 5, containerisation has been a blessing (currently Solr 7.7.2). In
> > our
> > > case we run 20 Solr nodes per host over 5 hosts totalling 100 Solr
> > > instances. Here I host 3 collections of varying size. The first
> contains
> > > 60m docs (8 shards), the second 360m (12 shards) , and the third 1.3b
> (30
> > > shards) all with 2 NRT replicas. The docs are primarily database
> sourced
> > > but are not tiny by any means.
> > >
> > > Here are some of my comments from our migration journey:
> > > - Running Solr on Docker should be no different to bare metal. You
> still
> > > need to test for your environment and conditions and follow the guides
> > and
> > > best practices outlined in the excellent Lucidworks blog post
> > >
> >
> https://lucidworks.com/post/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
> > > .
> > > - The recent Solr Docker images are built with Java 11 so if you store
> > > your indexes in hdfs you'll have to build your own Docker image as
> Hadoop
> > > is not yet certified with Java 11 (or use an older Solr version image
> > built
> > >

Re: Solr Cloud on Docker?

[Dwane Hall]

Hey Dominique,

>From a memory management perspective I don't do any container resource 
>limiting specifically in Docker (although as you mention you certainly can).  
>In our circumstances these hosts are used specifically for Solr so I planned 
>and tested my capacity beforehand. We have ~768G of RAM on each of these 5 
>hosts so with 20x16G heaps we had ~320G of heap being used by Solr, some 
>overhead for Docker and the other OS services leaving ~400G for the OS cache 
>and whatever wants to grab it on each host. Not everyone will have servers 
>this large which is why we really had to take advantage of multiple Solr 
>instances/host and Docker became important for our cluster operation 
>management.  Our disk's are not SSD's either and all instances write to the 
>same raid 5 spinner which is bind mounted to the containers.  With this 
>configuration we've been able to achieve consistent median response times of 
>under 500ms across the largest collection but obviously query type varies this 
>(no terms, leading wildcards etc.).  Our QPS is not huge ranging from 2-20/sec 
>but if we need to scale further or speed up response times there's certainly 
>wins that can be made at a disk level.  For our current circumstances we're 
>very content with the deployment.

In not sure if you've read Toke's blog on his experiences at the Royal Danish 
Library but I found it really useful when capacity planning and recommend 
reading it 
(https://sbdevel.wordpress.com/2016/11/30/70tb-16b-docs-4-machines-1-solrcloud/).

As always it's recommend to test for your own conditions and best of luck with 
your deployment!

Dwane


From: Scott Stults 
Sent: Thursday, 30 January 2020 1:45 AM
To: solr-user@lucene.apache.org 
Subject: Re: Solr Cloud on Docker?

One of our clients has been running a big Solr Cloud (100-ish nodes, TB
index, billions of docs) in kubernetes for over a year and it's been
wonderful. I think during that time the biggest scrapes we got were when we
ran out of disk space. Performance and reliability has been solid
otherwise. Like Dwane alluded to, a lot of operations pitfalls can be
avoided if you do your Docker orchestration through kubernetes.


k/r,
Scott

On Tue, Jan 28, 2020 at 3:34 AM Dominique Bejean 
wrote:

> Hi  Dwane,
>
> Thank you for sharing this great solr/docker user story.
>
> According to your Solr/JVM memory requirements (Heap size + MetaSpace +
> OffHeap size) are you specifying specific settings in docker-compose files
> (mem_limit, mem_reservation, mem_swappiness, ...) ?
> I suppose you are limiting total memory used by all dockerised Solr in
> order to keep free memory on host for MMAPDirectory ?
>
> In short can you explain the memory management ?
>
> Regards
>
> Dominique
>
>
>
>
> Le lun. 23 déc. 2019 à 00:17, Dwane Hall  a écrit :
>
> > Hey Walter,
> >
> > I recently migrated our Solr cluster to Docker and am very pleased I did
> > so. We run relativity large servers and run multiple Solr instances per
> > physical host and having managed Solr upgrades on bare metal installs
> since
> > Solr 5, containerisation has been a blessing (currently Solr 7.7.2). In
> our
> > case we run 20 Solr nodes per host over 5 hosts totalling 100 Solr
> > instances. Here I host 3 collections of varying size. The first contains
> > 60m docs (8 shards), the second 360m (12 shards) , and the third 1.3b (30
> > shards) all with 2 NRT replicas. The docs are primarily database sourced
> > but are not tiny by any means.
> >
> > Here are some of my comments from our migration journey:
> > - Running Solr on Docker should be no different to bare metal. You still
> > need to test for your environment and conditions and follow the guides
> and
> > best practices outlined in the excellent Lucidworks blog post
> >
> https://lucidworks.com/post/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
> > .
> > - The recent Solr Docker images are built with Java 11 so if you store
> > your indexes in hdfs you'll have to build your own Docker image as Hadoop
> > is not yet certified with Java 11 (or use an older Solr version image
> built
> > with Java 8)
> > - As Docker will be responsible for quite a few Solr nodes it becomes
> > important to make sure the Docker daemon is configured in systemctl to
> > restart after failure or reboot of the host. Additionally the Docker
> > restart=always setting is useful for restarting failed containers
> > automatically if a single container dies (i.e. JVM explosions). I've
> > deliberately blown up the JVM in test conditions and found the
> > containers/Solr recover really well under Docker.
> > - I use Docker Compose to s

Re: Solr Cloud on Docker?

[Scott Stults]

One of our clients has been running a big Solr Cloud (100-ish nodes, TB
index, billions of docs) in kubernetes for over a year and it's been
wonderful. I think during that time the biggest scrapes we got were when we
ran out of disk space. Performance and reliability has been solid
otherwise. Like Dwane alluded to, a lot of operations pitfalls can be
avoided if you do your Docker orchestration through kubernetes.


k/r,
Scott

On Tue, Jan 28, 2020 at 3:34 AM Dominique Bejean 
wrote:

> Hi  Dwane,
>
> Thank you for sharing this great solr/docker user story.
>
> According to your Solr/JVM memory requirements (Heap size + MetaSpace +
> OffHeap size) are you specifying specific settings in docker-compose files
> (mem_limit, mem_reservation, mem_swappiness, ...) ?
> I suppose you are limiting total memory used by all dockerised Solr in
> order to keep free memory on host for MMAPDirectory ?
>
> In short can you explain the memory management ?
>
> Regards
>
> Dominique
>
>
>
>
> Le lun. 23 déc. 2019 à 00:17, Dwane Hall  a écrit :
>
> > Hey Walter,
> >
> > I recently migrated our Solr cluster to Docker and am very pleased I did
> > so. We run relativity large servers and run multiple Solr instances per
> > physical host and having managed Solr upgrades on bare metal installs
> since
> > Solr 5, containerisation has been a blessing (currently Solr 7.7.2). In
> our
> > case we run 20 Solr nodes per host over 5 hosts totalling 100 Solr
> > instances. Here I host 3 collections of varying size. The first contains
> > 60m docs (8 shards), the second 360m (12 shards) , and the third 1.3b (30
> > shards) all with 2 NRT replicas. The docs are primarily database sourced
> > but are not tiny by any means.
> >
> > Here are some of my comments from our migration journey:
> > - Running Solr on Docker should be no different to bare metal. You still
> > need to test for your environment and conditions and follow the guides
> and
> > best practices outlined in the excellent Lucidworks blog post
> >
> https://lucidworks.com/post/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
> > .
> > - The recent Solr Docker images are built with Java 11 so if you store
> > your indexes in hdfs you'll have to build your own Docker image as Hadoop
> > is not yet certified with Java 11 (or use an older Solr version image
> built
> > with Java 8)
> > - As Docker will be responsible for quite a few Solr nodes it becomes
> > important to make sure the Docker daemon is configured in systemctl to
> > restart after failure or reboot of the host. Additionally the Docker
> > restart=always setting is useful for restarting failed containers
> > automatically if a single container dies (i.e. JVM explosions). I've
> > deliberately blown up the JVM in test conditions and found the
> > containers/Solr recover really well under Docker.
> > - I use Docker Compose to spin up our environment and it has been
> > excellent for maintaining consistent settings across Solr nodes and
> hosts.
> > Additionally using a .env file makes most of the Solr environment
> variables
> > per node configurable in an external file.
> > - I'd recommend Docker Swarm if you plan on running Solr over multiple
> > physical hosts. Unfortunately we had an incompatible OS so I was unable
> to
> > utilise this approach. The same incompatibility existed for K8s but
> > Lucidworks has another great article on this approach if you're more
> > fortunate with your environment than us
> > https://lucidworks.com/post/running-solr-on-kubernetes-part-1/.
> > - Our Solr instances are TLS secured and use the basic auth plugin and
> > rules based authentication provider. There's nothing I have not been able
> > to configure with the default Docker images using environment variables
> > passed into the container. This makes upgrades to Solr versions really
> easy
> > as you just need to grab the image and pass in your environment details
> to
> > the container for any new Solr version.
> > - If possible I'd start with the Solr 8 Docker image. The project
> > underwent a large refactor to align it with the install script based on
> > community feedback. If you start with an earlier version you'll need to
> > refactor when you eventually move to Solr version 8. The Solr Docker page
> > has more details on this.
> > - Matijn Koster (the project lead) is excellent and very responsive to
> > questions on the project page. Read through the q page before reaching
> > out I found a lot of my questions already answered there.  Additionally,
> he
> > provides a number of example Docker configurations from command line
> > parameters to docker-compose files running multiple instances and
> zookeeper
> > quarums.
> > - The Docker extra hosts parameter is useful for adding extra hosts to
> > your containers hosts file particularly if you have multiple nic cards
> with
> > internal and external interfaces and you want to force communication
> over a
> > specific one.
> > - We use the Solr 

Re: Solr Cloud on Docker?

[Dominique Bejean]

Hi  Dwane,

Thank you for sharing this great solr/docker user story.

According to your Solr/JVM memory requirements (Heap size + MetaSpace +
OffHeap size) are you specifying specific settings in docker-compose files
(mem_limit, mem_reservation, mem_swappiness, ...) ?
I suppose you are limiting total memory used by all dockerised Solr in
order to keep free memory on host for MMAPDirectory ?

In short can you explain the memory management ?

Regards

Dominique




Le lun. 23 déc. 2019 à 00:17, Dwane Hall  a écrit :

> Hey Walter,
>
> I recently migrated our Solr cluster to Docker and am very pleased I did
> so. We run relativity large servers and run multiple Solr instances per
> physical host and having managed Solr upgrades on bare metal installs since
> Solr 5, containerisation has been a blessing (currently Solr 7.7.2). In our
> case we run 20 Solr nodes per host over 5 hosts totalling 100 Solr
> instances. Here I host 3 collections of varying size. The first contains
> 60m docs (8 shards), the second 360m (12 shards) , and the third 1.3b (30
> shards) all with 2 NRT replicas. The docs are primarily database sourced
> but are not tiny by any means.
>
> Here are some of my comments from our migration journey:
> - Running Solr on Docker should be no different to bare metal. You still
> need to test for your environment and conditions and follow the guides and
> best practices outlined in the excellent Lucidworks blog post
> https://lucidworks.com/post/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
> .
> - The recent Solr Docker images are built with Java 11 so if you store
> your indexes in hdfs you'll have to build your own Docker image as Hadoop
> is not yet certified with Java 11 (or use an older Solr version image built
> with Java 8)
> - As Docker will be responsible for quite a few Solr nodes it becomes
> important to make sure the Docker daemon is configured in systemctl to
> restart after failure or reboot of the host. Additionally the Docker
> restart=always setting is useful for restarting failed containers
> automatically if a single container dies (i.e. JVM explosions). I've
> deliberately blown up the JVM in test conditions and found the
> containers/Solr recover really well under Docker.
> - I use Docker Compose to spin up our environment and it has been
> excellent for maintaining consistent settings across Solr nodes and hosts.
> Additionally using a .env file makes most of the Solr environment variables
> per node configurable in an external file.
> - I'd recommend Docker Swarm if you plan on running Solr over multiple
> physical hosts. Unfortunately we had an incompatible OS so I was unable to
> utilise this approach. The same incompatibility existed for K8s but
> Lucidworks has another great article on this approach if you're more
> fortunate with your environment than us
> https://lucidworks.com/post/running-solr-on-kubernetes-part-1/.
> - Our Solr instances are TLS secured and use the basic auth plugin and
> rules based authentication provider. There's nothing I have not been able
> to configure with the default Docker images using environment variables
> passed into the container. This makes upgrades to Solr versions really easy
> as you just need to grab the image and pass in your environment details to
> the container for any new Solr version.
> - If possible I'd start with the Solr 8 Docker image. The project
> underwent a large refactor to align it with the install script based on
> community feedback. If you start with an earlier version you'll need to
> refactor when you eventually move to Solr version 8. The Solr Docker page
> has more details on this.
> - Matijn Koster (the project lead) is excellent and very responsive to
> questions on the project page. Read through the q page before reaching
> out I found a lot of my questions already answered there.  Additionally, he
> provides a number of example Docker configurations from command line
> parameters to docker-compose files running multiple instances and zookeeper
> quarums.
> - The Docker extra hosts parameter is useful for adding extra hosts to
> your containers hosts file particularly if you have multiple nic cards with
> internal and external interfaces and you want to force communication over a
> specific one.
> - We use the Solr Prometheus exporter to collect node metrics. I've found
> I've needed to reduce the metrics to collect as having this many nodes
> overwhelmed it occasionally. From memory it had something to do with
> concurrent modification of Future objects the collector users and it
> sometimes misses collection cycles. This is not Docker related but Solr
> size related and the exporter's ability to handle it.
> - We use the zkCli script a lot for updating configsets. As I did not want
> to have to copy them into a container to update them I just download a copy
> of the Solr binaries and use it entirely for this zookeeper script. It's
> not elegant but a number of our Dev's are 

Re: Solr Cloud on Docker?

[Dwane Hall]

Hey Walter,

I recently migrated our Solr cluster to Docker and am very pleased I did so. We 
run relativity large servers and run multiple Solr instances per physical host 
and having managed Solr upgrades on bare metal installs since Solr 5, 
containerisation has been a blessing (currently Solr 7.7.2). In our case we run 
20 Solr nodes per host over 5 hosts totalling 100 Solr instances. Here I host 3 
collections of varying size. The first contains 60m docs (8 shards), the second 
360m (12 shards) , and the third 1.3b (30 shards) all with 2 NRT replicas. The 
docs are primarily database sourced but are not tiny by any means.

Here are some of my comments from our migration journey:
- Running Solr on Docker should be no different to bare metal. You still need 
to test for your environment and conditions and follow the guides and best 
practices outlined in the excellent Lucidworks blog post 
https://lucidworks.com/post/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/.
- The recent Solr Docker images are built with Java 11 so if you store your 
indexes in hdfs you'll have to build your own Docker image as Hadoop is not yet 
certified with Java 11 (or use an older Solr version image built with Java 8)
- As Docker will be responsible for quite a few Solr nodes it becomes important 
to make sure the Docker daemon is configured in systemctl to restart after 
failure or reboot of the host. Additionally the Docker restart=always setting 
is useful for restarting failed containers automatically if a single container 
dies (i.e. JVM explosions). I've deliberately blown up the JVM in test 
conditions and found the containers/Solr recover really well under Docker.
- I use Docker Compose to spin up our environment and it has been excellent for 
maintaining consistent settings across Solr nodes and hosts. Additionally using 
a .env file makes most of the Solr environment variables per node configurable 
in an external file.
- I'd recommend Docker Swarm if you plan on running Solr over multiple physical 
hosts. Unfortunately we had an incompatible OS so I was unable to utilise this 
approach. The same incompatibility existed for K8s but Lucidworks has another 
great article on this approach if you're more fortunate with your environment 
than us https://lucidworks.com/post/running-solr-on-kubernetes-part-1/.
- Our Solr instances are TLS secured and use the basic auth plugin and rules 
based authentication provider. There's nothing I have not been able to 
configure with the default Docker images using environment variables passed 
into the container. This makes upgrades to Solr versions really easy as you 
just need to grab the image and pass in your environment details to the 
container for any new Solr version.
- If possible I'd start with the Solr 8 Docker image. The project underwent a 
large refactor to align it with the install script based on community feedback. 
If you start with an earlier version you'll need to refactor when you 
eventually move to Solr version 8. The Solr Docker page has more details on 
this.
- Matijn Koster (the project lead) is excellent and very responsive to 
questions on the project page. Read through the q page before reaching out I 
found a lot of my questions already answered there.  Additionally, he provides 
a number of example Docker configurations from command line parameters to 
docker-compose files running multiple instances and zookeeper quarums.
- The Docker extra hosts parameter is useful for adding extra hosts to your 
containers hosts file particularly if you have multiple nic cards with internal 
and external interfaces and you want to force communication over a specific one.
- We use the Solr Prometheus exporter to collect node metrics. I've found I've 
needed to reduce the metrics to collect as having this many nodes overwhelmed 
it occasionally. From memory it had something to do with concurrent 
modification of Future objects the collector users and it sometimes misses 
collection cycles. This is not Docker related but Solr size related and the 
exporter's ability to handle it.
- We use the zkCli script a lot for updating configsets. As I did not want to 
have to copy them into a container to update them I just download a copy of the 
Solr binaries and use it entirely for this zookeeper script. It's not elegant 
but a number of our Dev's are not familiar with Docker and this was a nice 
compromise. Another alternative is to just use the rest API to do any configset 
manipulation.
- We load balance all of these nodes to external clients using a haproxy Docker 
image. This combined with the Docker restart policy and Solr replication and 
autoscaling capabilities provides a very stable environment for us.

All in all migrating and running Solr on Docker has been brilliant. It was 
primarily driven by a need to scale our environment vertically on large 
hardware instances as running 100 nodes on bare metal was too big a maintenance 
and 

Re: Solr Cloud Using Docker

[Vincenzo D'Amore]

Hi,

I did this https://github.com/freedev/solrcloud-zookeeper-docker
If you want give it a try, this project aims to help developers and newbies
that would try Solr Cloud and Zookeeper in a Docker environment.




On Fri, Sep 16, 2016 at 11:11 PM, John Bickerstaff  wrote:

> In case this is helpful - sponsored by Lucidworks
>
> https://hub.docker.com/_/solr/
>
> I can't speak to the pros and cons of using it in production except to say
> that they are probably the same as the pros and cons of running Docker in
> production (as in nothing is perfect)
>
> HTH...
>
> On Fri, Sep 16, 2016 at 2:37 PM, Brendan Grainger 
> wrote:
>
> > Hi,
> >
> > Does anyone used docker for deploying solr? I am using it for running a
> > single solr server ‘cloud’ locally on my dev box, but wondering about the
> > pros/cons of using it in production.
> >
> > Thanks,
> > Brendan
>



-- 
Vincenzo D'Amore
email: v.dam...@gmail.com
skype: free.dev
mobile: +39 349 8513251

Re: Solr Cloud Using Docker

[John Bickerstaff]

In case this is helpful - sponsored by Lucidworks

https://hub.docker.com/_/solr/

I can't speak to the pros and cons of using it in production except to say
that they are probably the same as the pros and cons of running Docker in
production (as in nothing is perfect)

HTH...

On Fri, Sep 16, 2016 at 2:37 PM, Brendan Grainger 
wrote:

> Hi,
>
> Does anyone used docker for deploying solr? I am using it for running a
> single solr server ‘cloud’ locally on my dev box, but wondering about the
> pros/cons of using it in production.
>
> Thanks,
> Brendan