Hi,
I'm seeing this random Authentication failure in our Solr Cloud cluster
which is eventually rendering the nodes in "down" state. This doesn't seem
to have a pattern, just starts to happen out of the blue. I've 2 shards,
each having two replicas. They are using Solr basic authentication plugin.
Here's the error log:
org.apache.solr.security.RuleBasedAuthorizationPlugin.checkPathPerm(RuleBasedAuthorizationPlugin.java:147)
- request has come without principal. failed permission {
"name":"select",
"collection":"knowledge",
"path":"/select",
"role":[
"admin",
"dev",
"read"]}
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:500) -
USER_REQUIRED auth header null context : userPrincipal: [null] type:
[READ], collections: [knowledge,], Path: [/select] path : /select params
:q=*:*&distrib=false&sort=_docid_+asc&rows=0&wt=javabin&version=2
It eventually throws zookeeper timeout session and disappears from the
cluster.
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1156) - *Client
session timed out, have not heard from server in 229984ms for sessionid
0x35ec984bea00016, closing socket connection and attempting reconnect*
If I restart the node, it goes into recovery mode, but at the same time,
the other healthy replica starts throwing the authentication error and
eventually spirals into the downed state. This happens across all the nodes
till everyone has gone through one restart cycle.
Here are a couple of other exceptions I've seen in the log:
org.apache.solr.handler.ReplicationHandler$DirectoryFileStream.write(ReplicationHandler.java:1539)
- Exception while writing response for params:
generation=14327&qt=/replication&file=_1cww.fdt&offset=127926272&checksum=true&wt=filestream&command=filecontent
java.io.IOException: java.util.concurrent.TimeoutException: *Idle timeout
expired: 50001/50000 ms*
at
org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:220)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:491)
at
org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:90)
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:213)
at
org.apache.solr.common.util.FastOutputStream.write(FastOutputStream.java:83)
at
org.apache.solr.handler.ReplicationHandler$DirectoryFileStream.write(ReplicationHandler.java:1520)
at org.apache.solr.core.SolrCore$3.write(SolrCore.java:2601)
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49)
at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:809)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:538)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
org.apache.solr.security.PKIAuthenticationPlugin.parseCipher(PKIAuthenticationPlugin.java:175)
- *Decryption failed , key must be wrong*
java.security.InvalidKeyException: No installed provider supports this key:
(null)
at javax.crypto.Cipher.chooseProvider(Cipher.java:893)
at javax.crypto.Cipher.init(Cipher.java:1249)
at javax.crypto.Cipher.init(Cipher.java:1186)
at org.apache.solr.util.CryptoKeys.decryptRSA(CryptoKeys.java:277)
at
org.apache.solr.security.PKIAuthenticationPlugin.parseCipher(PKIAuthenticationPlugin.java:173)
at
org.apache.solr.security.PKIAuthenticationPlugin.decipherHeader(PKIAuthenticationPlugin.java:160)
at
org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:118)
at
org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:430)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
