Re: Solr Cloud on Docker?

[Dwane Hall]

Hey Walter,

I recently migrated our Solr cluster to Docker and am very pleased I did so. We 
run relativity large servers and run multiple Solr instances per physical host 
and having managed Solr upgrades on bare metal installs since Solr 5, 
containerisation has been a blessing (currently Solr 7.7.2). In our case we run 
20 Solr nodes per host over 5 hosts totalling 100 Solr instances. Here I host 3 
collections of varying size. The first contains 60m docs (8 shards), the second 
360m (12 shards) , and the third 1.3b (30 shards) all with 2 NRT replicas. The 
docs are primarily database sourced but are not tiny by any means.

Here are some of my comments from our migration journey:
- Running Solr on Docker should be no different to bare metal. You still need 
to test for your environment and conditions and follow the guides and best 
practices outlined in the excellent Lucidworks blog post 
https://lucidworks.com/post/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/.
- The recent Solr Docker images are built with Java 11 so if you store your 
indexes in hdfs you'll have to build your own Docker image as Hadoop is not yet 
certified with Java 11 (or use an older Solr version image built with Java 8)
- As Docker will be responsible for quite a few Solr nodes it becomes important 
to make sure the Docker daemon is configured in systemctl to restart after 
failure or reboot of the host. Additionally the Docker restart=always setting 
is useful for restarting failed containers automatically if a single container 
dies (i.e. JVM explosions). I've deliberately blown up the JVM in test 
conditions and found the containers/Solr recover really well under Docker.
- I use Docker Compose to spin up our environment and it has been excellent for 
maintaining consistent settings across Solr nodes and hosts. Additionally using 
a .env file makes most of the Solr environment variables per node configurable 
in an external file.
- I'd recommend Docker Swarm if you plan on running Solr over multiple physical 
hosts. Unfortunately we had an incompatible OS so I was unable to utilise this 
approach. The same incompatibility existed for K8s but Lucidworks has another 
great article on this approach if you're more fortunate with your environment 
than us https://lucidworks.com/post/running-solr-on-kubernetes-part-1/.
- Our Solr instances are TLS secured and use the basic auth plugin and rules 
based authentication provider. There's nothing I have not been able to 
configure with the default Docker images using environment variables passed 
into the container. This makes upgrades to Solr versions really easy as you 
just need to grab the image and pass in your environment details to the 
container for any new Solr version.
- If possible I'd start with the Solr 8 Docker image. The project underwent a 
large refactor to align it with the install script based on community feedback. 
If you start with an earlier version you'll need to refactor when you 
eventually move to Solr version 8. The Solr Docker page has more details on 
this.
- Matijn Koster (the project lead) is excellent and very responsive to 
questions on the project page. Read through the q page before reaching out I 
found a lot of my questions already answered there.  Additionally, he provides 
a number of example Docker configurations from command line parameters to 
docker-compose files running multiple instances and zookeeper quarums.
- The Docker extra hosts parameter is useful for adding extra hosts to your 
containers hosts file particularly if you have multiple nic cards with internal 
and external interfaces and you want to force communication over a specific one.
- We use the Solr Prometheus exporter to collect node metrics. I've found I've 
needed to reduce the metrics to collect as having this many nodes overwhelmed 
it occasionally. From memory it had something to do with concurrent 
modification of Future objects the collector users and it sometimes misses 
collection cycles. This is not Docker related but Solr size related and the 
exporter's ability to handle it.
- We use the zkCli script a lot for updating configsets. As I did not want to 
have to copy them into a container to update them I just download a copy of the 
Solr binaries and use it entirely for this zookeeper script. It's not elegant 
but a number of our Dev's are not familiar with Docker and this was a nice 
compromise. Another alternative is to just use the rest API to do any configset 
manipulation.
- We load balance all of these nodes to external clients using a haproxy Docker 
image. This combined with the Docker restart policy and Solr replication and 
autoscaling capabilities provides a very stable environment for us.

All in all migrating and running Solr on Docker has been brilliant. It was 
primarily driven by a need to scale our environment vertically on large 
hardware instances as running 100 nodes on bare metal was too big a maintenance 
and 

Re: MoreLikeThis does not work

[Jörn Franke]

It looks like you are trying to do a more like this of all documents in the 
collection. I am not sure if this makes sense. Maybe you should put a query 
that results in less results, eg one that returns a specific document. 

> Am 22.12.2019 um 13:40 schrieb Nehemia Litterat :
> 
> Hi,
> Any help will be appreciated
> 
> Using solr 8.2
> 
> I created a core using
> 
> ./solr create -c
> 
> 
> imported example docs using
> 
> * curl 'http://localhost:32576/solr/playground/update?commit=true
> ' --data-binary
> @/home/vagrant/books.csv -H 'Content-type:application/csv'*
> 
> the data is the sample data coming with solr
> 
> I did not change anything else in the configuration
> 
> trying to add *mlt=true=name* to my query did not yield any result
> 
> *http://localhost:8983/solr/playground/select?mlt.fl=name=true=*%3A*
> *
> 
> 
> I always get
> 
> "moreLikeThis":{ "0553573403":{"numFound":0,"start":0,"docs":[] }}
> 
> in the end of my results
> 
> this is the full result
> 
> {
> "responseHeader":{ "status":0, "QTime":3, "params":{ "q":"*:*", "mlt":"true",
> "mlt.fl":"name", "_":"1577018250160"}}, "response":{"numFound":10,"start":0
> ,"docs":[ { "id":"0553573403", "cat":["book"], "name":["A Game of Thrones"],
> "price":[7.99], "inStock":[true], "author":["George R.R. Martin"], "series_t
> ":"A Song of Ice and Fire", "sequence_i":1, "genre_s":"fantasy", "_version_
> ":1653437692236529664}, { "id":"0553579908", "cat":["book"], "name":["A
> Clash of Kings"], "price":[7.99], "inStock":[true], "author":["George R.R.
> Martin"], "series_t":"A Song of Ice and Fire", "sequence_i":2, "genre_s":
> "fantasy", "_version_":1653437692242821120}, { "id":"055357342X", "cat":[
> "book"], "name":["A Storm of Swords"], "price":[7.99], "inStock":[true], "
> author":["George R.R. Martin"], "series_t":"A Song of Ice and Fire", "
> sequence_i":3, "genre_s":"fantasy", "_version_":1653437692243869696}, { "id
> ":"0553293354", "cat":["book"], "name":["Foundation"], "price":[7.99], "
> inStock":[true], "author":["Isaac Asimov"], "series_t":"Foundation Novels",
> "sequence_i":1, "genre_s":"scifi", "_version_":1653437692244918272}, { "id":
> "0812521390", "cat":["book"], "name":["The Black Company"], "price":[6.99],
> "inStock":[false], "author":["Glen Cook"], "series_t":"The Chronicles of
> The Black Company", "sequence_i":1, "genre_s":"fantasy", "_version_":
> 1653437692244918273}, { "id":"0812550706", "cat":["book"], "name":["Ender's
> Game"], "price":[6.99], "inStock":[true], "author":["Orson Scott Card"], "
> series_t":"Ender", "sequence_i":1, "genre_s":"scifi", "_version_":
> 1653437692245966848}, { "id":"0441385532", "cat":["book"], "name":["Jhereg"],
> "price":[7.95], "inStock":[false], "author":["Steven Brust"], "series_t":"Vlad
> Taltos", "sequence_i":1, "genre_s":"fantasy", "_version_":
> 1653437692245966849}, { "id":"0380014300", "cat":["book"], "name":["Nine
> Princes In Amber"], "price":[6.99], "inStock":[true], "author":["Roger
> Zelazny"], "series_t":"the Chronicles of Amber", "sequence_i":1, "genre_s":
> "fantasy", "_version_":1653437692247015424}, { "id":"0805080481", "cat":[
> "book"], "name":["The Book of Three"], "price":[5.99], "inStock":[true], "
> author":["Lloyd Alexander"], "series_t":"The Chronicles of Prydain", "
> sequence_i":1, "genre_s":"fantasy", "_version_":1653437692248064000}, { "id
> ":"080508049X", "cat":["book"], "name":["The Black Cauldron"], "price":[5.99
> ], "inStock":[true], "author":["Lloyd Alexander"], "series_t":"The
> Chronicles of Prydain", "sequence_i":2, "genre_s":"fantasy", "_version_":
> 1653437692248064001}] }, "moreLikeThis":{ "0553573403":{"numFound":0,"start
> ":0,"docs":[] }, "0553579908":{"numFound":0,"start":0,"docs":[] }, "
> 055357342X":{"numFound":0,"start":0,"docs":[] }, "0553293354":{"numFound":0
> ,"start":0,"docs":[] }, "0812521390":{"numFound":0,"start":0,"docs":[] }, "
> 0812550706":{"numFound":0,"start":0,"docs":[] }, "0441385532":{"numFound":0
> ,"start":0,"docs":[] }, "0380014300":{"numFound":0,"start":0,"docs":[] }, "
> 0805080481":{"numFound":0,"start":0,"docs":[] }, "080508049X":{"numFound":0
> ,"start":0,"docs":[] }}}
> 
> 
> 
> [image: photo]
> *Nehemia Litterat*
> Our story
> 
> +972-54-6609351 | nlitte...@gmail.com
> 
> Skype: nlitterat <#SignatureSanitizer_SafeHtmlFilter_>
> 
> Please consider your environmental responsibility. Before printing this
> e-mail message, ask yourself whether you really need a hard copy.
> Create your own email signature
> 

MoreLikeThis does not work

[Nehemia Litterat]

Hi,
Any help will be appreciated

Using solr 8.2

I created a core using

./solr create -c


imported example docs using

* curl 'http://localhost:32576/solr/playground/update?commit=true
' --data-binary
@/home/vagrant/books.csv -H 'Content-type:application/csv'*

the data is the sample data coming with solr

I did not change anything else in the configuration

trying to add *mlt=true=name* to my query did not yield any result

*http://localhost:8983/solr/playground/select?mlt.fl=name=true=*%3A*
*


I always get

"moreLikeThis":{ "0553573403":{"numFound":0,"start":0,"docs":[] }}

in the end of my results

this is the full result

 {
"responseHeader":{ "status":0, "QTime":3, "params":{ "q":"*:*", "mlt":"true",
"mlt.fl":"name", "_":"1577018250160"}}, "response":{"numFound":10,"start":0
,"docs":[ { "id":"0553573403", "cat":["book"], "name":["A Game of Thrones"],
"price":[7.99], "inStock":[true], "author":["George R.R. Martin"], "series_t
":"A Song of Ice and Fire", "sequence_i":1, "genre_s":"fantasy", "_version_
":1653437692236529664}, { "id":"0553579908", "cat":["book"], "name":["A
Clash of Kings"], "price":[7.99], "inStock":[true], "author":["George R.R.
Martin"], "series_t":"A Song of Ice and Fire", "sequence_i":2, "genre_s":
"fantasy", "_version_":1653437692242821120}, { "id":"055357342X", "cat":[
"book"], "name":["A Storm of Swords"], "price":[7.99], "inStock":[true], "
author":["George R.R. Martin"], "series_t":"A Song of Ice and Fire", "
sequence_i":3, "genre_s":"fantasy", "_version_":1653437692243869696}, { "id
":"0553293354", "cat":["book"], "name":["Foundation"], "price":[7.99], "
inStock":[true], "author":["Isaac Asimov"], "series_t":"Foundation Novels",
"sequence_i":1, "genre_s":"scifi", "_version_":1653437692244918272}, { "id":
"0812521390", "cat":["book"], "name":["The Black Company"], "price":[6.99],
"inStock":[false], "author":["Glen Cook"], "series_t":"The Chronicles of
The Black Company", "sequence_i":1, "genre_s":"fantasy", "_version_":
1653437692244918273}, { "id":"0812550706", "cat":["book"], "name":["Ender's
Game"], "price":[6.99], "inStock":[true], "author":["Orson Scott Card"], "
series_t":"Ender", "sequence_i":1, "genre_s":"scifi", "_version_":
1653437692245966848}, { "id":"0441385532", "cat":["book"], "name":["Jhereg"],
"price":[7.95], "inStock":[false], "author":["Steven Brust"], "series_t":"Vlad
Taltos", "sequence_i":1, "genre_s":"fantasy", "_version_":
1653437692245966849}, { "id":"0380014300", "cat":["book"], "name":["Nine
Princes In Amber"], "price":[6.99], "inStock":[true], "author":["Roger
Zelazny"], "series_t":"the Chronicles of Amber", "sequence_i":1, "genre_s":
"fantasy", "_version_":1653437692247015424}, { "id":"0805080481", "cat":[
"book"], "name":["The Book of Three"], "price":[5.99], "inStock":[true], "
author":["Lloyd Alexander"], "series_t":"The Chronicles of Prydain", "
sequence_i":1, "genre_s":"fantasy", "_version_":1653437692248064000}, { "id
":"080508049X", "cat":["book"], "name":["The Black Cauldron"], "price":[5.99
], "inStock":[true], "author":["Lloyd Alexander"], "series_t":"The
Chronicles of Prydain", "sequence_i":2, "genre_s":"fantasy", "_version_":
1653437692248064001}] }, "moreLikeThis":{ "0553573403":{"numFound":0,"start
":0,"docs":[] }, "0553579908":{"numFound":0,"start":0,"docs":[] }, "
055357342X":{"numFound":0,"start":0,"docs":[] }, "0553293354":{"numFound":0
,"start":0,"docs":[] }, "0812521390":{"numFound":0,"start":0,"docs":[] }, "
0812550706":{"numFound":0,"start":0,"docs":[] }, "0441385532":{"numFound":0
,"start":0,"docs":[] }, "0380014300":{"numFound":0,"start":0,"docs":[] }, "
0805080481":{"numFound":0,"start":0,"docs":[] }, "080508049X":{"numFound":0
,"start":0,"docs":[] }}}



[image: photo]
*Nehemia Litterat*
Our story

+972-54-6609351 | nlitte...@gmail.com

Skype: nlitterat <#SignatureSanitizer_SafeHtmlFilter_>

Please consider your environmental responsibility. Before printing this
e-mail message, ask yourself whether you really need a hard copy.
Create your own email signature