[
https://issues.apache.org/jira/browse/SOLR-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207115#comment-15207115
]
Noble Paul commented on SOLR-8327:
----------------------------------
Smart caching work flow is as follows
# a request for a collection comes to solrj
# look up in local cache . If not available read from ZK and populate local
cache
# make a request to the server optimistically assuming that the data in cache
is latest. But send extra information as a request parameter
(\_stateVer_=<coll-name>:<zk-version-of-state.json-in-cache>)
# at server. if this parameter is present, check locally if the version is
correct . If this node serves this collection, it always has the latest state.
So no ZK lookup is necessary
# If the version at server is newer send the latest version of the state as a
part of the payload
# solrj looks for this extra info in the payload. If there is no extra info,
the state it has is the latest. nothing needs to be done. If the payload
contains versions of the state, it means that the local version is stale,
invalidate the cache
> SolrDispatchFilter is not caching new state format, which results in live
> fetch from ZK per request if node does not contain core from collection
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-8327
> URL: https://issues.apache.org/jira/browse/SOLR-8327
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 5.3
> Reporter: Jessica Cheng Mallet
> Assignee: Varun Thacker
> Labels: solrcloud
>
> While perf testing with non-solrj client (request can be sent to any solr
> node), we noticed a huge amount of data from Zookeeper in our tcpdump (~1G
> for 20 second dump). From the thread dump, we noticed this:
> java.lang.Object.wait (Native Method)
> java.lang.Object.wait (Object.java:503)
> org.apache.zookeeper.ClientCnxn.submitRequest (ClientCnxn.java:1309)
> org.apache.zookeeper.ZooKeeper.getData (ZooKeeper.java:1152)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:345)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation
> (ZkCmdExecutor.java:61)
> org.apache.solr.common.cloud.SolrZkClient.getData (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> (ZkStateReader.java:841)
> org.apache.solr.common.cloud.ZkStateReader$7.get (ZkStateReader.java:515)
> org.apache.solr.common.cloud.ClusterState.getCollectionOrNull
> (ClusterState.java:175)
> org.apache.solr.common.cloud.ClusterState.getLeader (ClusterState.java:98)
> org.apache.solr.servlet.HttpSolrCall.getCoreByCollection
> (HttpSolrCall.java:784)
> org.apache.solr.servlet.HttpSolrCall.init (HttpSolrCall.java:272)
> org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:417)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter
> (SolrDispatchFilter.java:210)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter
> (SolrDispatchFilter.java:179)
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter
> (ServletHandler.java:1652)
> org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143)
> org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577)
> org.eclipse.jetty.server.session.SessionHandler.doHandle
> (SessionHandler.java:223)
> org.eclipse.jetty.server.handler.ContextHandler.doHandle
> (ContextHandler.java:1127)
> org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515)
> org.eclipse.jetty.server.session.SessionHandler.doScope
> (SessionHandler.java:185)
> org.eclipse.jetty.server.handler.ContextHandler.doScope
> (ContextHandler.java:1061)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141)
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle
> (ContextHandlerCollection.java:215)
> org.eclipse.jetty.server.handler.HandlerCollection.handle
> (HandlerCollection.java:110)
> org.eclipse.jetty.server.handler.HandlerWrapper.handle
> (HandlerWrapper.java:97)
> org.eclipse.jetty.server.Server.handle (Server.java:499)
> org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310)
> org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257)
> org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540)
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
> (QueuedThreadPool.java:635)
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run
> (QueuedThreadPool.java:555)
> java.lang.Thread.run (Thread.java:745)
> Looks like SolrDispatchFilter doesn't have caching similar to the
> collectionStateCache in CloudSolrClient, so if the node doesn't know about a
> collection in the new state format, it just live-fetch it from Zookeeper on
> every request.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
