Solr Cloud on Docker?

[Walter Underwood]

Does anyone have experience running a big Solr Cloud cluster on Docker 
containers? By “big”, I mean 35 million docs, 40 nodes, 8 shards, with 36 CPU 
instances. We are running version 6.6.2 right now, but could upgrade.

If people have specific things to do or avoid, I’d really appreciate it.

I got a couple of responses on the Slack channel, but I’d love more stories 
from the trenches. This is a direction for our company architecture.

We have a master/slave cluster (Solr 4.10.4) that is awesome. I can absolutely 
see running the slaves as containers. For Solr Cloud? Makes me nervous.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

RE: Solr8 changes how security.json restricts access to GUI

[Oakley, Craig (NIH/NLM/NCBI) [C]]

Thanks for the clarification

Created SOLR-14083


-Original Message-
From: Erick Erickson  
Sent: Friday, December 13, 2019 6:26 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr8 changes how security.json restricts access to GUI

Anyone who has an account can open a JIRA, have you created one?

> On Dec 13, 2019, at 5:10 PM, Oakley, Craig (NIH/NLM/NCBI) [C] 
>  wrote:
> 
> It looks as though I do not have an option under 
> issues.apache.org/jira/projects/SOLR/issues by which to create an issue. 
> Could you create one (and let me know its number)?
> 
> Thanks
> 
> -Original Message-
> From: Jan Høydahl  
> Sent: Friday, December 13, 2019 3:52 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr8 changes how security.json restricts access to GUI
> 
> Ok, se should perhaps print a warning somewhere that IE is not supported. Can 
> you file a JIRA issue? 
> 
> Jan Høydahl
> 
>> 13. des. 2019 kl. 21:43 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
>> :
>> 
>> Well that is progress: indeed Firefox and Chrome and Edge do indeed prompt 
>> for login and password (as desired). It is Internet Explorer which does not, 
>> nor does curl (that is to say, if you ask curl only to go to the top level: 
>> host:port/solr -- going any further it will complain, such as your 
>> /solr/admin/info/system example gets Error 401 Authentication failed, 
>> Response code: 401)
>> 
>> 
>> 
>> -Original Message-
>> From: Jan Høydahl  
>> Sent: Friday, December 13, 2019 2:15 PM
>> To: solr-user 
>> Subject: Re: Solr8 changes how security.json restricts access to GUI
>> 
>> I got your screenshot 
>> (https://www.dropbox.com/s/7tbn7gx3uag6jcg/crippledSolrGUI.jpg?dl=0 
>> )
>> 
>> This is quite uncommon. You should see a loging screen if you have basicAuth 
>> enabled.
>> Have you tried a different browser?
>> 
>> What do you get if you run this command
>> 
>> curl -i http://your-solr-url/solr/admin/info/system
>> 
>> Or if you use your browser’s developer tools to inspect network traffic?
>> 
>> Jan
>> 
>>> 12. des. 2019 kl. 23:49 skrev Jan Høydahl :
>>> 
>>> Attachments are stripped from list, can you post a link to the screenshot 
>>> of the UI when you first visit?
>>> 
>>> Jan
>>> 
> 12. des. 2019 kl. 17:27 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
> :
 
 Below is the security.json (with password hashes redacted): in Solr7.4 it 
 prompts for a password and (if you get it right) lets you into the whole 
 GUI; But in Solr8.1.1 and in Solr 8.3, it does not prompt for a password 
 before letting you into a crippled version of the GUI (as depicted in the 
 attachment)
 
 {
 "authentication":{
 "class":"solr.BasicAuthPlugin",
 "credentials":{
   "solradmin":"[redacted]",
   "pysolrmon":"[redacted]",
   "solrtrg":"[redacted]"},
 "":{"v":2}},
 "authorization":{
 "class":"solr.RuleBasedAuthorizationPlugin",
 "user-role":{
   "solradmin":[
 "admin",
 "allgen",
 "trgadmin",
 "genadmin"],
   "solrtrg":[
 "trgadmin",
 "allgen"],
   "pysolrmon":["clustatus_role"]},
 "permissions":[
   {
 "name":"gen_admin",
 "collection":"NULL",
 "path":"/admin/cores",
 "params":{"action":[
 "REGEX:(?i)CREATE",
 "REGEX:(?i)RENAME",
 "REGEX:(?i)SWAP",
 "REGEX:(?i)UNLOAD",
 "REGEX:(?i)SPLIT"]},
 "role":"genadmin"},
   {
 "name":"col_admin",
 "collection":null,
 "path":"/admin/collections",
 "params":{"action":[
 "REGEX:(?i)CREATE",
 "REGEX:(?i)MODIFYCOLLECTION",
 "REGEX:(?i)SPLITSHARD",
 "REGEX:(?i)CREATESHARD",
 "REGEX:(?i)DELETESHARD",
 "REGEX:(?i)CREATEALIAS",
 "REGEX:(?i)DELETEALIAS",
 "REGEX:(?i)DELETE",
 "REGEX:(?i)DELETEREPLICA",
 "REGEX:(?i)ADDREPLICA",
 "REGEX:(?i)CLUSTERPROP",
 "REGEX:(?i)MIGRATE",
 "REGEX:(?i)ADDROLE",
 "REGEX:(?i)REMOVEROLE",
 "REGEX:(?i)ADDREPLICAPROP",
 "REGEX:(?i)DELETEREPLICAPROP",
 "REGEX:(?i)BALANCESHARDUNIQUE",
 "REGEX:(?i)REBALANCELEADERS",
 "REGEX:(?i)FORCELEADER",
 "REGEX:(?i)MIGRATESTATEFORMAT"]},
 "role":"genadmin"},
   {
 "name":"security-edit",
 "role":"admin"},
   {
 "name":"clustatus",
 "path":"/admin/collections",
 "params":{"action":["REGEX:(?i)CLUSTERSTATUS"]},
 "role":[
   "clustatus_role",
   "allgen"],
 "collection":null},
   {
 "name":"corestatus",
 "path":"/admin/cores",
 "params":{"action":["REGEX:(?i)STATUS"]},
 "role":[
   "allgen",
   "clustatus_role"],
 

Re: Solr8 changes how security.json restricts access to GUI

[Erick Erickson]

Anyone who has an account can open a JIRA, have you created one?

> On Dec 13, 2019, at 5:10 PM, Oakley, Craig (NIH/NLM/NCBI) [C] 
>  wrote:
> 
> It looks as though I do not have an option under 
> issues.apache.org/jira/projects/SOLR/issues by which to create an issue. 
> Could you create one (and let me know its number)?
> 
> Thanks
> 
> -Original Message-
> From: Jan Høydahl  
> Sent: Friday, December 13, 2019 3:52 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr8 changes how security.json restricts access to GUI
> 
> Ok, se should perhaps print a warning somewhere that IE is not supported. Can 
> you file a JIRA issue? 
> 
> Jan Høydahl
> 
>> 13. des. 2019 kl. 21:43 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
>> :
>> 
>> Well that is progress: indeed Firefox and Chrome and Edge do indeed prompt 
>> for login and password (as desired). It is Internet Explorer which does not, 
>> nor does curl (that is to say, if you ask curl only to go to the top level: 
>> host:port/solr -- going any further it will complain, such as your 
>> /solr/admin/info/system example gets Error 401 Authentication failed, 
>> Response code: 401)
>> 
>> 
>> 
>> -Original Message-
>> From: Jan Høydahl  
>> Sent: Friday, December 13, 2019 2:15 PM
>> To: solr-user 
>> Subject: Re: Solr8 changes how security.json restricts access to GUI
>> 
>> I got your screenshot 
>> (https://www.dropbox.com/s/7tbn7gx3uag6jcg/crippledSolrGUI.jpg?dl=0 
>> )
>> 
>> This is quite uncommon. You should see a loging screen if you have basicAuth 
>> enabled.
>> Have you tried a different browser?
>> 
>> What do you get if you run this command
>> 
>> curl -i http://your-solr-url/solr/admin/info/system
>> 
>> Or if you use your browser’s developer tools to inspect network traffic?
>> 
>> Jan
>> 
>>> 12. des. 2019 kl. 23:49 skrev Jan Høydahl :
>>> 
>>> Attachments are stripped from list, can you post a link to the screenshot 
>>> of the UI when you first visit?
>>> 
>>> Jan
>>> 
> 12. des. 2019 kl. 17:27 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
> :
 
 Below is the security.json (with password hashes redacted): in Solr7.4 it 
 prompts for a password and (if you get it right) lets you into the whole 
 GUI; But in Solr8.1.1 and in Solr 8.3, it does not prompt for a password 
 before letting you into a crippled version of the GUI (as depicted in the 
 attachment)
 
 {
 "authentication":{
 "class":"solr.BasicAuthPlugin",
 "credentials":{
   "solradmin":"[redacted]",
   "pysolrmon":"[redacted]",
   "solrtrg":"[redacted]"},
 "":{"v":2}},
 "authorization":{
 "class":"solr.RuleBasedAuthorizationPlugin",
 "user-role":{
   "solradmin":[
 "admin",
 "allgen",
 "trgadmin",
 "genadmin"],
   "solrtrg":[
 "trgadmin",
 "allgen"],
   "pysolrmon":["clustatus_role"]},
 "permissions":[
   {
 "name":"gen_admin",
 "collection":"NULL",
 "path":"/admin/cores",
 "params":{"action":[
 "REGEX:(?i)CREATE",
 "REGEX:(?i)RENAME",
 "REGEX:(?i)SWAP",
 "REGEX:(?i)UNLOAD",
 "REGEX:(?i)SPLIT"]},
 "role":"genadmin"},
   {
 "name":"col_admin",
 "collection":null,
 "path":"/admin/collections",
 "params":{"action":[
 "REGEX:(?i)CREATE",
 "REGEX:(?i)MODIFYCOLLECTION",
 "REGEX:(?i)SPLITSHARD",
 "REGEX:(?i)CREATESHARD",
 "REGEX:(?i)DELETESHARD",
 "REGEX:(?i)CREATEALIAS",
 "REGEX:(?i)DELETEALIAS",
 "REGEX:(?i)DELETE",
 "REGEX:(?i)DELETEREPLICA",
 "REGEX:(?i)ADDREPLICA",
 "REGEX:(?i)CLUSTERPROP",
 "REGEX:(?i)MIGRATE",
 "REGEX:(?i)ADDROLE",
 "REGEX:(?i)REMOVEROLE",
 "REGEX:(?i)ADDREPLICAPROP",
 "REGEX:(?i)DELETEREPLICAPROP",
 "REGEX:(?i)BALANCESHARDUNIQUE",
 "REGEX:(?i)REBALANCELEADERS",
 "REGEX:(?i)FORCELEADER",
 "REGEX:(?i)MIGRATESTATEFORMAT"]},
 "role":"genadmin"},
   {
 "name":"security-edit",
 "role":"admin"},
   {
 "name":"clustatus",
 "path":"/admin/collections",
 "params":{"action":["REGEX:(?i)CLUSTERSTATUS"]},
 "role":[
   "clustatus_role",
   "allgen"],
 "collection":null},
   {
 "name":"corestatus",
 "path":"/admin/cores",
 "params":{"action":["REGEX:(?i)STATUS"]},
 "role":[
   "allgen",
   "clustatus_role"],
 "collection":null},
   {
 "name":"trgadmin",
 "collection":"trg_col",
 "path":"/admin/*",
 "role":"trgadmin"},
   {
 "name":"open_select",
 "path":"/select/*",
 "role":null},
 

RE: Solr8 changes how security.json restricts access to GUI

[Oakley, Craig (NIH/NLM/NCBI) [C]]

It looks as though I do not have an option under 
issues.apache.org/jira/projects/SOLR/issues by which to create an issue. Could 
you create one (and let me know its number)?

Thanks

-Original Message-
From: Jan Høydahl  
Sent: Friday, December 13, 2019 3:52 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr8 changes how security.json restricts access to GUI

Ok, se should perhaps print a warning somewhere that IE is not supported. Can 
you file a JIRA issue? 

Jan Høydahl

> 13. des. 2019 kl. 21:43 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
> :
> 
> Well that is progress: indeed Firefox and Chrome and Edge do indeed prompt 
> for login and password (as desired). It is Internet Explorer which does not, 
> nor does curl (that is to say, if you ask curl only to go to the top level: 
> host:port/solr -- going any further it will complain, such as your 
> /solr/admin/info/system example gets Error 401 Authentication failed, 
> Response code: 401)
> 
> 
> 
> -Original Message-
> From: Jan Høydahl  
> Sent: Friday, December 13, 2019 2:15 PM
> To: solr-user 
> Subject: Re: Solr8 changes how security.json restricts access to GUI
> 
> I got your screenshot 
> (https://www.dropbox.com/s/7tbn7gx3uag6jcg/crippledSolrGUI.jpg?dl=0 
> )
> 
> This is quite uncommon. You should see a loging screen if you have basicAuth 
> enabled.
> Have you tried a different browser?
> 
> What do you get if you run this command
> 
> curl -i http://your-solr-url/solr/admin/info/system
> 
> Or if you use your browser’s developer tools to inspect network traffic?
> 
> Jan
> 
>> 12. des. 2019 kl. 23:49 skrev Jan Høydahl :
>> 
>> Attachments are stripped from list, can you post a link to the screenshot of 
>> the UI when you first visit?
>> 
>> Jan
>> 
 12. des. 2019 kl. 17:27 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
 :
>>> 
>>> Below is the security.json (with password hashes redacted): in Solr7.4 it 
>>> prompts for a password and (if you get it right) lets you into the whole 
>>> GUI; But in Solr8.1.1 and in Solr 8.3, it does not prompt for a password 
>>> before letting you into a crippled version of the GUI (as depicted in the 
>>> attachment)
>>> 
>>> {
>>> "authentication":{
>>>  "class":"solr.BasicAuthPlugin",
>>>  "credentials":{
>>>"solradmin":"[redacted]",
>>>"pysolrmon":"[redacted]",
>>>"solrtrg":"[redacted]"},
>>>  "":{"v":2}},
>>> "authorization":{
>>>  "class":"solr.RuleBasedAuthorizationPlugin",
>>>  "user-role":{
>>>"solradmin":[
>>>  "admin",
>>>  "allgen",
>>>  "trgadmin",
>>>  "genadmin"],
>>>"solrtrg":[
>>>  "trgadmin",
>>>  "allgen"],
>>>"pysolrmon":["clustatus_role"]},
>>>  "permissions":[
>>>{
>>>  "name":"gen_admin",
>>>  "collection":"NULL",
>>>  "path":"/admin/cores",
>>>  "params":{"action":[
>>>  "REGEX:(?i)CREATE",
>>>  "REGEX:(?i)RENAME",
>>>  "REGEX:(?i)SWAP",
>>>  "REGEX:(?i)UNLOAD",
>>>  "REGEX:(?i)SPLIT"]},
>>>  "role":"genadmin"},
>>>{
>>>  "name":"col_admin",
>>>  "collection":null,
>>>  "path":"/admin/collections",
>>>  "params":{"action":[
>>>  "REGEX:(?i)CREATE",
>>>  "REGEX:(?i)MODIFYCOLLECTION",
>>>  "REGEX:(?i)SPLITSHARD",
>>>  "REGEX:(?i)CREATESHARD",
>>>  "REGEX:(?i)DELETESHARD",
>>>  "REGEX:(?i)CREATEALIAS",
>>>  "REGEX:(?i)DELETEALIAS",
>>>  "REGEX:(?i)DELETE",
>>>  "REGEX:(?i)DELETEREPLICA",
>>>  "REGEX:(?i)ADDREPLICA",
>>>  "REGEX:(?i)CLUSTERPROP",
>>>  "REGEX:(?i)MIGRATE",
>>>  "REGEX:(?i)ADDROLE",
>>>  "REGEX:(?i)REMOVEROLE",
>>>  "REGEX:(?i)ADDREPLICAPROP",
>>>  "REGEX:(?i)DELETEREPLICAPROP",
>>>  "REGEX:(?i)BALANCESHARDUNIQUE",
>>>  "REGEX:(?i)REBALANCELEADERS",
>>>  "REGEX:(?i)FORCELEADER",
>>>  "REGEX:(?i)MIGRATESTATEFORMAT"]},
>>>  "role":"genadmin"},
>>>{
>>>  "name":"security-edit",
>>>  "role":"admin"},
>>>{
>>>  "name":"clustatus",
>>>  "path":"/admin/collections",
>>>  "params":{"action":["REGEX:(?i)CLUSTERSTATUS"]},
>>>  "role":[
>>>"clustatus_role",
>>>"allgen"],
>>>  "collection":null},
>>>{
>>>  "name":"corestatus",
>>>  "path":"/admin/cores",
>>>  "params":{"action":["REGEX:(?i)STATUS"]},
>>>  "role":[
>>>"allgen",
>>>"clustatus_role"],
>>>  "collection":null},
>>>{
>>>  "name":"trgadmin",
>>>  "collection":"trg_col",
>>>  "path":"/admin/*",
>>>  "role":"trgadmin"},
>>>{
>>>  "name":"open_select",
>>>  "path":"/select/*",
>>>  "role":null},
>>>{
>>>  "name":"open_search",
>>>  "path":"/search/*",
>>>  "role":null},
>>>{
>>>  "name":"catch-all-nocollection",
>>>  "collection":null,
>>>  "path":"/*",
>>>  "role":"allgen"},
>>>{
>>>

Re: Solr8 changes how security.json restricts access to GUI

[Jan Høydahl]

Ok, se should perhaps print a warning somewhere that IE is not supported. Can 
you file a JIRA issue? 

Jan Høydahl

> 13. des. 2019 kl. 21:43 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
> :
> 
> Well that is progress: indeed Firefox and Chrome and Edge do indeed prompt 
> for login and password (as desired). It is Internet Explorer which does not, 
> nor does curl (that is to say, if you ask curl only to go to the top level: 
> host:port/solr -- going any further it will complain, such as your 
> /solr/admin/info/system example gets Error 401 Authentication failed, 
> Response code: 401)
> 
> 
> 
> -Original Message-
> From: Jan Høydahl  
> Sent: Friday, December 13, 2019 2:15 PM
> To: solr-user 
> Subject: Re: Solr8 changes how security.json restricts access to GUI
> 
> I got your screenshot 
> (https://www.dropbox.com/s/7tbn7gx3uag6jcg/crippledSolrGUI.jpg?dl=0 
> )
> 
> This is quite uncommon. You should see a loging screen if you have basicAuth 
> enabled.
> Have you tried a different browser?
> 
> What do you get if you run this command
> 
> curl -i http://your-solr-url/solr/admin/info/system
> 
> Or if you use your browser’s developer tools to inspect network traffic?
> 
> Jan
> 
>> 12. des. 2019 kl. 23:49 skrev Jan Høydahl :
>> 
>> Attachments are stripped from list, can you post a link to the screenshot of 
>> the UI when you first visit?
>> 
>> Jan
>> 
 12. des. 2019 kl. 17:27 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
 :
>>> 
>>> Below is the security.json (with password hashes redacted): in Solr7.4 it 
>>> prompts for a password and (if you get it right) lets you into the whole 
>>> GUI; But in Solr8.1.1 and in Solr 8.3, it does not prompt for a password 
>>> before letting you into a crippled version of the GUI (as depicted in the 
>>> attachment)
>>> 
>>> {
>>> "authentication":{
>>>  "class":"solr.BasicAuthPlugin",
>>>  "credentials":{
>>>"solradmin":"[redacted]",
>>>"pysolrmon":"[redacted]",
>>>"solrtrg":"[redacted]"},
>>>  "":{"v":2}},
>>> "authorization":{
>>>  "class":"solr.RuleBasedAuthorizationPlugin",
>>>  "user-role":{
>>>"solradmin":[
>>>  "admin",
>>>  "allgen",
>>>  "trgadmin",
>>>  "genadmin"],
>>>"solrtrg":[
>>>  "trgadmin",
>>>  "allgen"],
>>>"pysolrmon":["clustatus_role"]},
>>>  "permissions":[
>>>{
>>>  "name":"gen_admin",
>>>  "collection":"NULL",
>>>  "path":"/admin/cores",
>>>  "params":{"action":[
>>>  "REGEX:(?i)CREATE",
>>>  "REGEX:(?i)RENAME",
>>>  "REGEX:(?i)SWAP",
>>>  "REGEX:(?i)UNLOAD",
>>>  "REGEX:(?i)SPLIT"]},
>>>  "role":"genadmin"},
>>>{
>>>  "name":"col_admin",
>>>  "collection":null,
>>>  "path":"/admin/collections",
>>>  "params":{"action":[
>>>  "REGEX:(?i)CREATE",
>>>  "REGEX:(?i)MODIFYCOLLECTION",
>>>  "REGEX:(?i)SPLITSHARD",
>>>  "REGEX:(?i)CREATESHARD",
>>>  "REGEX:(?i)DELETESHARD",
>>>  "REGEX:(?i)CREATEALIAS",
>>>  "REGEX:(?i)DELETEALIAS",
>>>  "REGEX:(?i)DELETE",
>>>  "REGEX:(?i)DELETEREPLICA",
>>>  "REGEX:(?i)ADDREPLICA",
>>>  "REGEX:(?i)CLUSTERPROP",
>>>  "REGEX:(?i)MIGRATE",
>>>  "REGEX:(?i)ADDROLE",
>>>  "REGEX:(?i)REMOVEROLE",
>>>  "REGEX:(?i)ADDREPLICAPROP",
>>>  "REGEX:(?i)DELETEREPLICAPROP",
>>>  "REGEX:(?i)BALANCESHARDUNIQUE",
>>>  "REGEX:(?i)REBALANCELEADERS",
>>>  "REGEX:(?i)FORCELEADER",
>>>  "REGEX:(?i)MIGRATESTATEFORMAT"]},
>>>  "role":"genadmin"},
>>>{
>>>  "name":"security-edit",
>>>  "role":"admin"},
>>>{
>>>  "name":"clustatus",
>>>  "path":"/admin/collections",
>>>  "params":{"action":["REGEX:(?i)CLUSTERSTATUS"]},
>>>  "role":[
>>>"clustatus_role",
>>>"allgen"],
>>>  "collection":null},
>>>{
>>>  "name":"corestatus",
>>>  "path":"/admin/cores",
>>>  "params":{"action":["REGEX:(?i)STATUS"]},
>>>  "role":[
>>>"allgen",
>>>"clustatus_role"],
>>>  "collection":null},
>>>{
>>>  "name":"trgadmin",
>>>  "collection":"trg_col",
>>>  "path":"/admin/*",
>>>  "role":"trgadmin"},
>>>{
>>>  "name":"open_select",
>>>  "path":"/select/*",
>>>  "role":null},
>>>{
>>>  "name":"open_search",
>>>  "path":"/search/*",
>>>  "role":null},
>>>{
>>>  "name":"catch-all-nocollection",
>>>  "collection":null,
>>>  "path":"/*",
>>>  "role":"allgen"},
>>>{
>>>  "name":"catch-all-collection",
>>>  "path":"/*",
>>>  "role":"allgen"},
>>>{
>>>  "name":"all-admincol",
>>>  "collection":null,
>>>  "path":"/admin/collections",
>>>  "role":"allgen"},
>>>{
>>>  "name":"all-admincores",
>>>  "collection":null,
>>>  "path":"/admin/cores",
>>>  "role":"allgen"}],
>>>  "":{"v":5}}}
>>> 

RE: Solr8 changes how security.json restricts access to GUI

[Oakley, Craig (NIH/NLM/NCBI) [C]]

Well that is progress: indeed Firefox and Chrome and Edge do indeed prompt for 
login and password (as desired). It is Internet Explorer which does not, nor 
does curl (that is to say, if you ask curl only to go to the top level: 
host:port/solr -- going any further it will complain, such as your 
/solr/admin/info/system example gets Error 401 Authentication failed, Response 
code: 401)



-Original Message-
From: Jan Høydahl  
Sent: Friday, December 13, 2019 2:15 PM
To: solr-user 
Subject: Re: Solr8 changes how security.json restricts access to GUI

I got your screenshot 
(https://www.dropbox.com/s/7tbn7gx3uag6jcg/crippledSolrGUI.jpg?dl=0 
)

This is quite uncommon. You should see a loging screen if you have basicAuth 
enabled.
Have you tried a different browser?

What do you get if you run this command

curl -i http://your-solr-url/solr/admin/info/system

Or if you use your browser’s developer tools to inspect network traffic?

Jan

> 12. des. 2019 kl. 23:49 skrev Jan Høydahl :
> 
> Attachments are stripped from list, can you post a link to the screenshot of 
> the UI when you first visit?
> 
> Jan
> 
>> 12. des. 2019 kl. 17:27 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
>> :
>> 
>> Below is the security.json (with password hashes redacted): in Solr7.4 it 
>> prompts for a password and (if you get it right) lets you into the whole 
>> GUI; But in Solr8.1.1 and in Solr 8.3, it does not prompt for a password 
>> before letting you into a crippled version of the GUI (as depicted in the 
>> attachment)
>> 
>> {
>> "authentication":{
>>   "class":"solr.BasicAuthPlugin",
>>   "credentials":{
>> "solradmin":"[redacted]",
>> "pysolrmon":"[redacted]",
>> "solrtrg":"[redacted]"},
>>   "":{"v":2}},
>> "authorization":{
>>   "class":"solr.RuleBasedAuthorizationPlugin",
>>   "user-role":{
>> "solradmin":[
>>   "admin",
>>   "allgen",
>>   "trgadmin",
>>   "genadmin"],
>> "solrtrg":[
>>   "trgadmin",
>>   "allgen"],
>> "pysolrmon":["clustatus_role"]},
>>   "permissions":[
>> {
>>   "name":"gen_admin",
>>   "collection":"NULL",
>>   "path":"/admin/cores",
>>   "params":{"action":[
>>   "REGEX:(?i)CREATE",
>>   "REGEX:(?i)RENAME",
>>   "REGEX:(?i)SWAP",
>>   "REGEX:(?i)UNLOAD",
>>   "REGEX:(?i)SPLIT"]},
>>   "role":"genadmin"},
>> {
>>   "name":"col_admin",
>>   "collection":null,
>>   "path":"/admin/collections",
>>   "params":{"action":[
>>   "REGEX:(?i)CREATE",
>>   "REGEX:(?i)MODIFYCOLLECTION",
>>   "REGEX:(?i)SPLITSHARD",
>>   "REGEX:(?i)CREATESHARD",
>>   "REGEX:(?i)DELETESHARD",
>>   "REGEX:(?i)CREATEALIAS",
>>   "REGEX:(?i)DELETEALIAS",
>>   "REGEX:(?i)DELETE",
>>   "REGEX:(?i)DELETEREPLICA",
>>   "REGEX:(?i)ADDREPLICA",
>>   "REGEX:(?i)CLUSTERPROP",
>>   "REGEX:(?i)MIGRATE",
>>   "REGEX:(?i)ADDROLE",
>>   "REGEX:(?i)REMOVEROLE",
>>   "REGEX:(?i)ADDREPLICAPROP",
>>   "REGEX:(?i)DELETEREPLICAPROP",
>>   "REGEX:(?i)BALANCESHARDUNIQUE",
>>   "REGEX:(?i)REBALANCELEADERS",
>>   "REGEX:(?i)FORCELEADER",
>>   "REGEX:(?i)MIGRATESTATEFORMAT"]},
>>   "role":"genadmin"},
>> {
>>   "name":"security-edit",
>>   "role":"admin"},
>> {
>>   "name":"clustatus",
>>   "path":"/admin/collections",
>>   "params":{"action":["REGEX:(?i)CLUSTERSTATUS"]},
>>   "role":[
>> "clustatus_role",
>> "allgen"],
>>   "collection":null},
>> {
>>   "name":"corestatus",
>>   "path":"/admin/cores",
>>   "params":{"action":["REGEX:(?i)STATUS"]},
>>   "role":[
>> "allgen",
>> "clustatus_role"],
>>   "collection":null},
>> {
>>   "name":"trgadmin",
>>   "collection":"trg_col",
>>   "path":"/admin/*",
>>   "role":"trgadmin"},
>> {
>>   "name":"open_select",
>>   "path":"/select/*",
>>   "role":null},
>> {
>>   "name":"open_search",
>>   "path":"/search/*",
>>   "role":null},
>> {
>>   "name":"catch-all-nocollection",
>>   "collection":null,
>>   "path":"/*",
>>   "role":"allgen"},
>> {
>>   "name":"catch-all-collection",
>>   "path":"/*",
>>   "role":"allgen"},
>> {
>>   "name":"all-admincol",
>>   "collection":null,
>>   "path":"/admin/collections",
>>   "role":"allgen"},
>> {
>>   "name":"all-admincores",
>>   "collection":null,
>>   "path":"/admin/cores",
>>   "role":"allgen"}],
>>   "":{"v":5}}}
>> 
>> -Original Message-
>> From: Jan Høydahl  
>> Sent: Wednesday, December 11, 2019 7:35 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr8 changes how security.json restricts access to GUI
>> 
>> Please show your complete Security.json so we know how auth is 

Re: Get Solr to notice new core without restarting?

[Mikhail Khludnev]

https://lucene.apache.org/solr/guide/8_2/coreadmin-api.html#coreadmin-create


On Fri, Dec 13, 2019 at 10:50 PM Mark H. Wood  wrote:

> I have a product which comes with several empty Solr cores already
> configured and laid out, ready to be copied into place where Solr can
> find them.  Is there a way to get Solr to notice new cores without
> restarting it?  Is it likely there ever will be?  I'm one of the
> people who test and maintain the product, so I'm always creating and
> destroying instances.
>
> --
> Mark H. Wood
> Lead Technology Analyst
>
> University Library
> Indiana University - Purdue University Indianapolis
> 755 W. Michigan Street
> Indianapolis, IN 46202
> 317-274-0749
> www.ulib.iupui.edu
>


-- 
Sincerely yours
Mikhail Khludnev

Get Solr to notice new core without restarting?

[Mark H. Wood]

I have a product which comes with several empty Solr cores already
configured and laid out, ready to be copied into place where Solr can
find them.  Is there a way to get Solr to notice new cores without
restarting it?  Is it likely there ever will be?  I'm one of the
people who test and maintain the product, so I'm always creating and
destroying instances.

-- 
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu


signature.asc
Description: PGP signature

unable to update using empty strings or 'null' in value

[rhys J]

When I do the following update:

curl http://localhost:8983/solr/dbtr/update?commit=true -d '[{ "id":
"601000", "agen
t": {"set": "null"},"assign_id": {"set": "320"},"client_group": {"set":
"null"},"credit_class": {"se
t": "null"},"credit_hold": {"set": "null"},"credit_hold_date": {"set":
"null"},"credit_limit": {"set
": "null"},"credit_terms": {"set": "null"},"currency": {"set":
"null"},"data_signature": {"set": "nu
ll"},"debtor_id": {"set": "601000"},"dl1": {"set": "25611"},"dl2": {"set":
"null"},"do_not_call": {"
set": "null"},"do_not_call_date": {"set": "null"},"do_not_report": {"set":
"null"},"in_aris_date": {
"set": "2016-09-01 00:00:00"},"name1": {"set": "60 Grit Studios"},"name2":
{"set": "null"},"next_con
tact_date": {"set": "2018-12-24 00:00:00"},"parent_customer_number":
{"set": "null"},"potential_bad_debt": {"set": "null"},"priority_followup":
{"set": "null"},"reference_no": {"set": "25611"},"report_as": {"set":
"null"},"report_status": {"set": "null"},"risk": {"set":
"null"},"rms_acct_id": {"set": "null"},"salesperson": {"set":
"null"},"ssn1": {"set": "null"},"ssn2": {"set": "null"},"status_code":
{"set": "172"},"status_date": {"set": "2018-10-30 00:00:00"},"timestamp":
{"set": "null"},"tz_offset": {"set": "null"},"warning_item_no": {"set":
"null"},"watch_list": {"set": "null"},"watch_list_date": {"set": "null"},}]'

I get the following error:

 "error":{
"msg":"For input string: \"null\"",
"trace":"java.lang.NumberFormatException: For input string:
\"null\"\n\tat java.base/jdk.interna
l.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)\n\tat
java.base/jdk.internal.
math.FloatingDecimal.parseFloat(FloatingDecimal.java:122)\n\tat
java.base/java.lang.Float.parseFloat
(Float.java:461)\n\tat
org.apache.solr.schema.IntPointField.toNativeType(IntPointField.java:54)\n\ta
t
org.apache.solr.update.processor.AtomicUpdateDocumentMerger.getNativeFieldValue(AtomicUpdateDocume
ntMerger.java:563)\n\tat
org.apache.solr.update.processor.AtomicUpdateDocumentMerger.doSet(AtomicUpd
ateDocumentMerger.java:436)\n\tat
org.apache.solr.update.processor.AtomicUpdateDocumentMerger.merge(
AtomicUpdateDocumentMerger.java:112)\n\tat
org.apache.solr.update.processor.DistributedUpdateProcess
or.getUpdatedDocument(DistributedUpdateProcessor.java:704)\n\tat
org.apache.solr.update.processor.Di
stributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:372)\n\tat
org.apache.solr.upd
ate.processor.DistributedUpdateProcessor.lambda$versionAdd$0(DistributedUpdateProcessor.java:337)\n\
tat
org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50)\n\tat
org.apache.solr.up
date.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:337)\n\tat
org.
apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:2
23)\n\tat
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(L
ogUpdateProcessorFactory.java:103)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)\n\tat
org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:475)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)\n\tat
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)\n\tat
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)\n\tat
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)\n\tat
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)\n\tat
org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:75)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)\n\tat
org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:92)\n\tat
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:507)\n\tat
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:145)\n\tat
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:121)\n\tat

Re: Solr8 changes how security.json restricts access to GUI

[Jan Høydahl]

I got your screenshot 
(https://www.dropbox.com/s/7tbn7gx3uag6jcg/crippledSolrGUI.jpg?dl=0 
)

This is quite uncommon. You should see a loging screen if you have basicAuth 
enabled.
Have you tried a different browser?

What do you get if you run this command

curl -i http://your-solr-url/solr/admin/info/system

Or if you use your browser’s developer tools to inspect network traffic?

Jan

> 12. des. 2019 kl. 23:49 skrev Jan Høydahl :
> 
> Attachments are stripped from list, can you post a link to the screenshot of 
> the UI when you first visit?
> 
> Jan
> 
>> 12. des. 2019 kl. 17:27 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
>> :
>> 
>> Below is the security.json (with password hashes redacted): in Solr7.4 it 
>> prompts for a password and (if you get it right) lets you into the whole 
>> GUI; But in Solr8.1.1 and in Solr 8.3, it does not prompt for a password 
>> before letting you into a crippled version of the GUI (as depicted in the 
>> attachment)
>> 
>> {
>> "authentication":{
>>   "class":"solr.BasicAuthPlugin",
>>   "credentials":{
>> "solradmin":"[redacted]",
>> "pysolrmon":"[redacted]",
>> "solrtrg":"[redacted]"},
>>   "":{"v":2}},
>> "authorization":{
>>   "class":"solr.RuleBasedAuthorizationPlugin",
>>   "user-role":{
>> "solradmin":[
>>   "admin",
>>   "allgen",
>>   "trgadmin",
>>   "genadmin"],
>> "solrtrg":[
>>   "trgadmin",
>>   "allgen"],
>> "pysolrmon":["clustatus_role"]},
>>   "permissions":[
>> {
>>   "name":"gen_admin",
>>   "collection":"NULL",
>>   "path":"/admin/cores",
>>   "params":{"action":[
>>   "REGEX:(?i)CREATE",
>>   "REGEX:(?i)RENAME",
>>   "REGEX:(?i)SWAP",
>>   "REGEX:(?i)UNLOAD",
>>   "REGEX:(?i)SPLIT"]},
>>   "role":"genadmin"},
>> {
>>   "name":"col_admin",
>>   "collection":null,
>>   "path":"/admin/collections",
>>   "params":{"action":[
>>   "REGEX:(?i)CREATE",
>>   "REGEX:(?i)MODIFYCOLLECTION",
>>   "REGEX:(?i)SPLITSHARD",
>>   "REGEX:(?i)CREATESHARD",
>>   "REGEX:(?i)DELETESHARD",
>>   "REGEX:(?i)CREATEALIAS",
>>   "REGEX:(?i)DELETEALIAS",
>>   "REGEX:(?i)DELETE",
>>   "REGEX:(?i)DELETEREPLICA",
>>   "REGEX:(?i)ADDREPLICA",
>>   "REGEX:(?i)CLUSTERPROP",
>>   "REGEX:(?i)MIGRATE",
>>   "REGEX:(?i)ADDROLE",
>>   "REGEX:(?i)REMOVEROLE",
>>   "REGEX:(?i)ADDREPLICAPROP",
>>   "REGEX:(?i)DELETEREPLICAPROP",
>>   "REGEX:(?i)BALANCESHARDUNIQUE",
>>   "REGEX:(?i)REBALANCELEADERS",
>>   "REGEX:(?i)FORCELEADER",
>>   "REGEX:(?i)MIGRATESTATEFORMAT"]},
>>   "role":"genadmin"},
>> {
>>   "name":"security-edit",
>>   "role":"admin"},
>> {
>>   "name":"clustatus",
>>   "path":"/admin/collections",
>>   "params":{"action":["REGEX:(?i)CLUSTERSTATUS"]},
>>   "role":[
>> "clustatus_role",
>> "allgen"],
>>   "collection":null},
>> {
>>   "name":"corestatus",
>>   "path":"/admin/cores",
>>   "params":{"action":["REGEX:(?i)STATUS"]},
>>   "role":[
>> "allgen",
>> "clustatus_role"],
>>   "collection":null},
>> {
>>   "name":"trgadmin",
>>   "collection":"trg_col",
>>   "path":"/admin/*",
>>   "role":"trgadmin"},
>> {
>>   "name":"open_select",
>>   "path":"/select/*",
>>   "role":null},
>> {
>>   "name":"open_search",
>>   "path":"/search/*",
>>   "role":null},
>> {
>>   "name":"catch-all-nocollection",
>>   "collection":null,
>>   "path":"/*",
>>   "role":"allgen"},
>> {
>>   "name":"catch-all-collection",
>>   "path":"/*",
>>   "role":"allgen"},
>> {
>>   "name":"all-admincol",
>>   "collection":null,
>>   "path":"/admin/collections",
>>   "role":"allgen"},
>> {
>>   "name":"all-admincores",
>>   "collection":null,
>>   "path":"/admin/cores",
>>   "role":"allgen"}],
>>   "":{"v":5}}}
>> 
>> -Original Message-
>> From: Jan Høydahl  
>> Sent: Wednesday, December 11, 2019 7:35 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr8 changes how security.json restricts access to GUI
>> 
>> Please show your complete Security.json so we know how auth is configured. 
>> Which 8.x version are you trying? There should be a login screen shown in 
>> admin UI now.
>> 
>> Jan Høydahl
>> 
>>> 11. des. 2019 kl. 22:40 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
>>> :
>>> 
>>> In Solr 7, we had clauses in our security.json saying
>>> 
>>>{
>>>  "name":"all-admin",
>>>  "collection":null,
>>>  "path":"/*",
>>>  "role":"allgen",
>>>  "index":15},
>>>{
>>>  "name":"all-core-handlers",
>>>  "path":"/*",
>>>  "role":"allgen",
>>>  "index":16},
>>> 
>>> We granted the role allgen to 

Re: Updates via curl and json not showing in api

[rhys J]

On Fri, Dec 13, 2019 at 11:51 AM Shawn Heisey  wrote:


> > Is there a step I'm missing?
>
> It appears that you have not executed a commit that opens a new searcher.
>
>
Thanks for explaining this.

I turned on commit=true, and everything works as expected.

Thanks again,

Rhys

Re: Updates via curl and json not showing in api

[Shawn Heisey]

On 12/13/2019 9:23 AM, rhys J wrote:

When I do the following update:

curl http://localhost:8983/solr/debt/update -d '[ {"id": "393291-18625",
"orig_int_amt":{ "set" : "2.5"}, }]'

and then:

curl http://localhost:8983/solr/debt/get?id=393291-18625

I see the document is updated via the command line.

It shows the following:



 "orig_int_amt":2.5,



 "orig_int_amt":0.0,



How do I get the updates to show in the API?

Is there a step I'm missing?


It appears that you have not executed a commit that opens a new searcher.

You can see the change in the realtime get handler (/get) which shows 
the latest version of documents, even those that have been indexed but 
aren't yet visible to normal searchers, but you can't see it when using 
the normal searcher.


https://lucene.apache.org/solr/guide/6_6/realtime-get.html

The simplest solution is to issue a commit and let Solr open a new searcher.

Example Solr configs are set up with autoCommit, using a 15 second 
maxTime, but in that config, openSearcher is set to false.  Which means 
that the automatic commits will NOT have their changes visible.  If you 
want changes to be automatically visible even if you never send an 
explicit commit, you'll need to set that up.


https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

That blog post says it is for SolrCloud, but its concepts are universal 
and do not depend on Solr running in cloud mode.


Thanks,
Shawn

Updates via curl and json not showing in api

[rhys J]

When I do the following update:

curl http://localhost:8983/solr/debt/update -d '[ {"id": "393291-18625",
"orig_int_amt":{ "set" : "2.5"}, }]'

and then:

curl http://localhost:8983/solr/debt/get?id=393291-18625

I see the document is updated via the command line.

It shows the following:

{
  "doc":
  {
"id":"393291-18625",
"adjust_int":0.0,
"adjust_princ":0.0,
"clt_id":"8032",
"clt_ref_no":"ORD15096 000",
"comments":" ",
"debt_descr":"PO/XREF: 904-132985337, SHIPPER: BERGER AUSTIN",
"debt_id":"393291",
"debt_no":18625,
"debt_type":"COM",
"delq_date":"2015-02-08T00:00:00Z",
"internal_adjustment":0,
"list_date":"2015-01-13T00:00:00Z",
"orig_clt":"8032",
"orig_int_amt":2.5,
"orig_princ_amt":49.3,
"potential_bad_debt":0,
"princ_paid":49.3,
"reference_no":"invoice:ORD15096 000",
"serv_date":"2015-01-09T00:00:00Z",
"status_code":520,
"status_date":"2015-02-20T00:00:00Z",
"storage_account":0,
"time_stamp":"2015-01-13T06:09:00Z",
"_version_":1652822026780409856}}

But when I use the Solr Web API, I get the following:

  {
"id":"393291-18625",
"adjust_int":0.0,
"adjust_princ":0.0,
"clt_id":"8032",
"clt_ref_no":"ORD15096 000",
"comments":" ",
"debt_descr":"PO/XREF: 904-132985337, SHIPPER: BERGER AUSTIN",
"debt_id":"393291",
"debt_no":18625,
"debt_type":"COM",
"delq_date":"2015-02-08T00:00:00Z",
"internal_adjustment":0,
"list_date":"2015-01-13T00:00:00Z",
"orig_clt":"8032",
"orig_int_amt":0.0,
"orig_princ_amt":49.3,
"potential_bad_debt":0,
"princ_paid":49.3,
"reference_no":"invoice:ORD15096 000",
"serv_date":"2015-01-09T00:00:00Z",
"status_code":520,
"status_date":"2015-02-20T00:00:00Z",
"storage_account":0,
"time_stamp":"2015-01-13T06:09:00Z",
"_version_":1652734816636895232},

Notice that orig_int_amt is still 0.0.

How do I get the updates to show in the API?

Is there a step I'm missing?

Thanks,

Rhys

Re: Sometimes searching slow in Solr 6.1.0

[Shawn Heisey]

On 12/13/2019 12:29 AM, vishal patel wrote:

We have 2 shards and 2 replicas in our live environment. Total of 26 
collections. we give 64GB RAM for a single Solr instance.


Are you saying that the machine has 64GB of memory, or that the Java 
heap for Solr is 64GB?


Looking over the list history, it looks like you've started other 
threads about performance problems, and I have advised you on some of 
them.  From what you had said on those threads, I thought you were on a 
much newer version than 6.1.0.


You've been given the following wiki page before.  It would be a good 
idea to read it fully, and come back with any questions you have about it.


https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue


I have faced a slow searching issue in our live environment. In our scenario, 
there are many update requests come within minutes like 50,000. At that time 
searching becomes slow.
The query is normal but taking a 4to5 seconds. When the same query will execute 
after sometimes it will not take time.
Our solr config details:
* autoCommit is 6 and autoSoftCommit is 100.


The maxTime values for those are in milliseconds.  So if you have 
maxTime for autoSoftCommit set to 100, that's a tenth of a second. 
There's no way that setting can be healthy.  The autoSoftCommit setting 
should be much longer.  I'd start at two minutes (12) and adjust 
from there.  If it's maxDocs that's set to 100, that is also way too 
small.  I would recommend only setting maxTime.


Thanks,
Shawn