setting up solr on tomcat
Hi, Is the solrTomcat wiki article valid for solr-4.7.0 ? http://wiki.apache.org/solr/SolrTomcat I am not able to deploy solr after following the instructions there. When I try to access the solr admin page I get a 404. I followed every step exactly as mentioned in the wiki, still no dice. Any ideas ? -- View this message in context: http://lucene.472066.n3.nabble.com/setting-up-solr-on-tomcat-tp4126177.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: using SolrJ with SolrCloud, searching multiple indexes.
Yeah sorry didn't explain myself there, one of the three zookeepers will return me one of the solrcloud machines for me to access the index. I either need to know which machine it returned(is this feasible I can't seem to find a way to access information in SolrCloudServer) and then add the extra indexes as shards String shards = solrCloudMachine+:8080/indexB,+solrCloudMachine+:8080/indexC (solrQuery.add(shards,shards);) or do it in a new way within solrcloud. FYI My returned index is one of seven indexes under one webapp (solr_search) I want to stitch on the other six indexes so I can search all of the data (each index is updated from separate feeds). Thanks for your quick reply. Russ. From: Furkan KAMACI [furkankam...@gmail.com] Sent: 21 March 2014 22:55 To: solr-user@lucene.apache.org Subject: Re: using SolrJ with SolrCloud, searching multiple indexes. Hi Russell; You say that: | CloudSolrServer server = new CloudSolrServer(solrServer1: 2111,solrServer2:2111,solrServer2:2111); but I should mention that they are not Solr Servers that is passed into a CloudSolrServer. They are zookeeper host:port pairs optionally includes a chroot parameter at the end. Thanks; Furkan KAMACI 2014-03-21 18:11 GMT+02:00 Russell Taylor russell.tay...@interactivedata.com: Hi, just started to move my SolrJ queries over to our SolrCloud environment and I want to know how to do a query where you combine multiple indexes. Previously I had a string called shards which links all the indexes together and adds them to the query. String shards = server:8080/solr_search/bonds,server:8080/solr_search/equities,etc which I add to my SolrQuery solrQuery.add(shards,shards); I can then search across many indexes. In SolrCloud we do this CloudSolrServer server = new CloudSolrServer(solrServer1:2111,solrServer2:2111,solrServer2:2111); and add the default collection server.setDefaultCollection(bonds); How do I add the other indexes to my query in CloudSolrServer? If it's as before solrQuery.add(shards,shards); how do I find out the address of the machine CloudSolrServer has chosen? Thanks Russ. *** This message (including any files transmitted with it) may contain confidential and/or proprietary information, is the property of Interactive Data Corporation and/or its subsidiaries, and is directed only to the addressee(s). If you are not the designated recipient or have reason to believe you received this message in error, please delete this message from your system and notify the sender immediately. An unintended recipient's disclosure, copying, distribution, or use of this message or any attachments is prohibited and may be unlawful. *** *** This message (including any files transmitted with it) may contain confidential and/or proprietary information, is the property of Interactive Data Corporation and/or its subsidiaries, and is directed only to the addressee(s). If you are not the designated recipient or have reason to believe you received this message in error, please delete this message from your system and notify the sender immediately. An unintended recipient's disclosure, copying, distribution, or use of this message or any attachments is prohibited and may be unlawful. ***
Re: Best approach to handle large volume of documents with constantly high incoming rate?
Any thoughts? Can Solr Cloud support such use case with acceptable performance? On Thursday, March 20, 2014 7:51 PM, shushuai zhu ss...@yahoo.com wrote: Hi, I am looking for some advice to handle large volume of documents with a very high incoming rate. The size of each document is about 0.5 KB and the incoming rate could be more than 20K per second and we want to store about one year's documents in Solr for near real=time searching. The goal is to achieve acceptable indexing and querying performance. We will use techniques like soft commit, dedicated indexing servers, etc. My main question is about how to structure the collection/shard/core to achieve the goals. Since the incoming rate is very high, we do not want the incoming documents to affect the existing older indexes. Some thoughts are to create a latest index to hold the incoming documents (say latest half hour's data, about 36M docs) so queries on older data could be faster since the old indexes are not affected. There seem three ways to grow the time dimension by adding/splitting/creating a new object listed below every half hour: collection shard core Which is the best way to grow the time dimension? Any limitation in that direction? Or there is some better approach? As an example, I am thinking about having 4 nodes with the following configuration to setup a Solr Cloud: Memory: 128 GB Storage: 4 TB How to set the collection/shard/core to deal with the use case? Thanks in advance. Shushuai
Re: Best approach to handle large volume of documents with constantly high incoming rate?
20K docs/sec = 20,000 * 60 * 60 * 24 = 1,728,000,000 = 1.7 billion docs/day * 365 = 630,720,000,000 = 631 billion docs/yr At 100 million docs/node = 6,308 nodes! And you think you can do it with 4 nodes? Oh, and that's before replication! 0.5K/doc * 631 billion docs = 322 TB. -- Jack Krupansky -Original Message- From: shushuai zhu Sent: Saturday, March 22, 2014 11:32 AM To: solr-user@lucene.apache.org Subject: Re: Best approach to handle large volume of documents with constantly high incoming rate? Any thoughts? Can Solr Cloud support such use case with acceptable performance? On Thursday, March 20, 2014 7:51 PM, shushuai zhu ss...@yahoo.com wrote: Hi, I am looking for some advice to handle large volume of documents with a very high incoming rate. The size of each document is about 0.5 KB and the incoming rate could be more than 20K per second and we want to store about one year's documents in Solr for near real=time searching. The goal is to achieve acceptable indexing and querying performance. We will use techniques like soft commit, dedicated indexing servers, etc. My main question is about how to structure the collection/shard/core to achieve the goals. Since the incoming rate is very high, we do not want the incoming documents to affect the existing older indexes. Some thoughts are to create a latest index to hold the incoming documents (say latest half hour's data, about 36M docs) so queries on older data could be faster since the old indexes are not affected. There seem three ways to grow the time dimension by adding/splitting/creating a new object listed below every half hour: collection shard core Which is the best way to grow the time dimension? Any limitation in that direction? Or there is some better approach? As an example, I am thinking about having 4 nodes with the following configuration to setup a Solr Cloud: Memory: 128 GB Storage: 4 TB How to set the collection/shard/core to deal with the use case? Thanks in advance. Shushuai
Solr Cloud collection keep going down?
We have 2 collections with 1 shard each replicated over 5 servers in the cluster. We see a lot of flapping (down or recovering) on one of the collections. When this happens the other collection hosted on the same machine is still marked as active. When this happens it takes a fairly long time (~30 minutes) for the collection to come back online, if at all. I find that its usually more reliable to completely shutdown solr on the affected machine and bring it back up with its core disabled. We then re-enable the core when its marked as active. A few questions: 1) What is the healthcheck in Solr-Cloud? Put another way, what is failing that marks one collection as down but the other on the same machine as up? 2) Why does recovery take forever when a node goes down.. even if its only down for 30 seconds. Our index is only 7-8G and we are running on SSD's. 3) What can be done to diagnose and fix this problem?
Re: Solr Cloud collection keep going down?
Some logs the core in question is items.
-
WARN - 2014-03-22 02:37:13.344;
org.apache.solr.cloud.RecoveryStrategy; Stopping recovery for
zkNodeName=10.0.14.101:8983_solr_itemscore=items
WARN - 2014-03-22 02:37:16.352; org.apache.solr.update.PeerSync;
PeerSync: core=items url=http://10.0.14.101:8983/solr too many updates
received since start - startingUpdates no longer overlaps with our
currentUpdates
WARN - 2014-03-22 02:46:26.277; org.apache.solr.core.SolrCore;
[items] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
WARN - 2014-03-22 02:49:27.736;
org.apache.solr.update.UpdateLog$LogReplayer; Starting log replay
tlog{file=/var/lib/solr-parent/solr-web/home/items/data/tlog/tlog.409
refcount=2} active=true starting pos=98026856
WARN - 2014-03-22 02:50:07.896; org.apache.solr.core.SolrCore;
[items] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
ERROR - 2014-03-22 02:51:49.640; org.apache.solr.common.SolrException;
null:org.eclipse.jetty.io.EofException
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)
at
org.eclipse.jetty.http.AbstractGenerator.blockForOutput(AbstractGenerator.java:507)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at org.apache.solr.util.FastWriter.flush(FastWriter.java:141)
at org.apache.solr.util.FastWriter.write(FastWriter.java:55)
at org.apache.solr.response.RubyWriter.writeStr(RubyResponseWriter.java:92)
at
org.apache.solr.response.JSONWriter.writeNamedListAsFlat(JSONResponseWriter.java:285)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:301)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:188)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:183)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:299)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:188)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:183)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:299)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:188)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:183)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:299)
at org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:95)
at org.apache.solr.response.RubyResponseWriter.write(RubyResponseWriter.java:37)
at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:768)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:440)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
Re: Solr4.7 No live SolrServers available to handle this request
Thanks Michael! I just committed your fix. It will be released with 4.7.1 On Fri, Mar 21, 2014 at 8:30 PM, Michael Sokolov msoko...@safaribooksonline.com wrote: I just managed to track this down -- as you said the disconnect was a red herring. Ultimately the problem was caused by a custom analysis component we wrote that was raising an IOException -- it was missing some configuration files it relies on. What might be interesting for solr devs to have a look at is that exception was completely swallowed by JavabinCodec, making it very difficult to track down the problem. Furthermore -- if the /add request was routed directly to the shard where the document was destined to end up, then the IOException raised by the analysis component (a char filter) showed up in the Solr HTTP response (probably because my client used XML format in one test -- javabin is used internally in SolrCloud). But if the request was routed to a different shard, then the only exception that showed up anywhere (in the logs, in the HTTP response) was kind of irrelevant. I think this could be fixed pretty easily; see SOLR-5985 for my suggestion. -Mike On 03/21/2014 10:20 AM, Greg Walters wrote: Broken pipe errors are generally caused by unexpected disconnections and are some times hard to track down. Given the stack traces you've provided it's hard to point to any one thing and I suspect the relevant information was snipped out in the long dump of document fields. You might grab the entire error from the client you're uploading documents with, the server you're connected to and any other nodes that have an error at the same time and put it on pastebin or the like. Thanks, Greg On Mar 20, 2014, at 3:36 PM, Michael Sokolov msoko...@safaribooksonline.com wrote: I'm getting a similar exception when writing documents (on the client side). I can write one document fine, but the second (which is being routed to a different shard) generates the error. It happens every time - definitely not a resource issue or timing problem since this database is completely empty -- I'm just getting started and running some tests, so there must be some kind of setup problem. But it's difficult to diagnose (for me, anyway)! I'd appreciate any insight, hints, guesses, etc. since I'm stuck. Thanks! One node (the leader?) is reporting Internal Server Error in its log, and another node (presumably the shard where the document is being directed) bombs out like this: ERROR - 2014-03-20 15:56:53.022; org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: ERROR adding document SolrInputDocument( ... long dump of document fields ) at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:99) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136) at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121) at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190) at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173) at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106) at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721) ... Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366) at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240) at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:119) at
Re: Limit on # of collections -SolrCloud
I figured out that most of the startup time seems to spent on waiting for replicas to recover. It waits from 6 seconds all the way upto 600 seconds for replicas to recover before trying again and sometimes it succeeds and otherwise it marks the core as down. Is there a way to reduce the timeout while recovery ? Also can anyone explain why the recovery takes so long ? Cant it mark itself as the leader and not wait for some replica to be available? *Logs*: ERROR - 2014-03-22 19:34:07.852; org.apache.solr.common.SolrException; Error while trying to recover. core=testcollection_shard5_replica1:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: I was asked to wait on state recovering for 10.1.1.100:8983_solr but I still do not see the requested state. I see state: active live:true at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:402) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:202) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:346) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223) ERROR - 2014-03-22 19:34:07.853; org.apache.solr.cloud.RecoveryStrategy; Recovery failed - trying again... (6) core= testcollection_shard5_replica1 INFO - 2014-03-22 19:34:07.853; org.apache.solr.cloud.RecoveryStrategy; Wait 128.0 seconds before trying to recover again (7) On Fri, Mar 21, 2014 at 1:05 PM, Chris W chris1980@gmail.com wrote: Sorry for the piecemeal approach but had another question. I have a 3 zk ensemble. Does making 2 zk as observer roles help speed up bootup of solr (due to decrease in time it takes to decide leaders for shards)? On Fri, Mar 21, 2014 at 11:49 AM, Chris W chris1980@gmail.com wrote: Thanks Tim. I would definitely try that next time. I have seen a few instances where the overseer_queue not getting processed but that looks like an existing bug which got fixed in 4.6 (overseer doesnt process requests when reload collection fails) One question: Assuming our cluster can tolerate downtime of about 10-15 minutes, is it ok to restart all solrnodes at the same time? or will there be race conditions while recovery? On Fri, Mar 21, 2014 at 11:08 AM, Mark Miller markrmil...@gmail.comwrote: On March 21, 2014 at 1:46:13 PM, Tim Potter (tim.pot...@lucidworks.com) wrote: We've seen instances where you end up restarting the overseer node each time as you restart the cluster, which causes all kinds of craziness. That would be a great test to add tot he suite. -- Mark Miller about.me/markrmiller -- Best -- C -- Best -- C -- Best -- C
Re: using SolrJ with SolrCloud, searching multiple indexes.
On 3/22/2014 7:34 AM, Russell Taylor wrote: Yeah sorry didn't explain myself there, one of the three zookeepers will return me one of the solrcloud machines for me to access the index. I either need to know which machine it returned(is this feasible I can't seem to find a way to access information in SolrCloudServer) and then add the extra indexes as shards String shards = solrCloudMachine+:8080/indexB,+solrCloudMachine+:8080/indexC (solrQuery.add(shards,shards);) or do it in a new way within solrcloud. FYI My returned index is one of seven indexes under one webapp (solr_search) I want to stitch on the other six indexes so I can search all of the data (each index is updated from separate feeds). SolrCloud eliminates the need to use the shards parameter, so CloudSolrServer does not expose the actual Solr instances. You *can* use the shards parameter, but typically it is done differently than traditional Solr. CloudSolrServer thinks mostly in terms of collections, not cores. There is a setDefaultCollection method on the server object, or you can do solrQuery.set(collection,name). You can query certain shards of a collection or multiple collections, without ever knowing the host/port/core combinations: http://wiki.apache.org/solr/SolrCloud#Distributed_Requests There are also collection aliases on the server side, which let you access one or more real collections with a virtual collection name. Thanks, Shawn
Re: Solr Cloud collection keep going down?
On 3/22/2014 1:23 PM, Software Dev wrote: We have 2 collections with 1 shard each replicated over 5 servers in the cluster. We see a lot of flapping (down or recovering) on one of the collections. When this happens the other collection hosted on the same machine is still marked as active. When this happens it takes a fairly long time (~30 minutes) for the collection to come back online, if at all. I find that its usually more reliable to completely shutdown solr on the affected machine and bring it back up with its core disabled. We then re-enable the core when its marked as active. A few questions: 1) What is the healthcheck in Solr-Cloud? Put another way, what is failing that marks one collection as down but the other on the same machine as up? 2) Why does recovery take forever when a node goes down.. even if its only down for 30 seconds. Our index is only 7-8G and we are running on SSD's. 3) What can be done to diagnose and fix this problem? Unless you are actually using the ping request handler, the healthcheck config will not matter. Or were you referring to something else? Referencing the logs you included in your reply: The EofException errors happen because your client code times out and disconnects before the request it made has completed. That is most likely just a symptom that has nothing at all to do with the problem. Read the following wiki page. What I'm going to say below will reference information you can find there: http://wiki.apache.org/solr/SolrPerformanceProblems Relevant side note: The default zookeeper client timeout is 15 seconds. A typical zookeeper config defines tickTime as 2 seconds, and the timeout cannot be configured to be more than 20 times the tickTime, which means it cannot go beyond 40 seconds. The default timeout value 15 seconds is usually more than enough, unless you are having performance problems. If you are not actually taking Solr instances down, then the fact that you are seeing the log replay messages indicates to me that something is taking so much time that the connection to Zookeeper times out. When it finally responds, it will attempt to recover the index, which means first it will replay the transaction log and then it might replicate the index from the shard leader. Replaying the transaction log is likely the reason it takes so long to recover. The wiki page I linked above has a slow startup section that explains how to fix this. There is some kind of underlying problem that is causing the zookeeper connection to timeout. It is most likely garbage collection pauses or insufficient RAM to cache the index, possibly both. You did not indicate how much total RAM you have or how big your Java heap is. As the wiki page mentions in the SSD section, SSD is not a substitute for having enough RAM to cache at significant percentage of your index. Thanks, Shawn
Re: Solr4.7 No live SolrServers available to handle this request
Excellent, thanks Shalin! On 3/22/2014 3:32 PM, Shalin Shekhar Mangar wrote: Thanks Michael! I just committed your fix. It will be released with 4.7.1 On Fri, Mar 21, 2014 at 8:30 PM, Michael Sokolov msoko...@safaribooksonline.com wrote: I just managed to track this down -- as you said the disconnect was a red herring. Ultimately the problem was caused by a custom analysis component we wrote that was raising an IOException -- it was missing some configuration files it relies on. What might be interesting for solr devs to have a look at is that exception was completely swallowed by JavabinCodec, making it very difficult to track down the problem. Furthermore -- if the /add request was routed directly to the shard where the document was destined to end up, then the IOException raised by the analysis component (a char filter) showed up in the Solr HTTP response (probably because my client used XML format in one test -- javabin is used internally in SolrCloud). But if the request was routed to a different shard, then the only exception that showed up anywhere (in the logs, in the HTTP response) was kind of irrelevant. I think this could be fixed pretty easily; see SOLR-5985 for my suggestion. -Mike On 03/21/2014 10:20 AM, Greg Walters wrote: Broken pipe errors are generally caused by unexpected disconnections and are some times hard to track down. Given the stack traces you've provided it's hard to point to any one thing and I suspect the relevant information was snipped out in the long dump of document fields. You might grab the entire error from the client you're uploading documents with, the server you're connected to and any other nodes that have an error at the same time and put it on pastebin or the like. Thanks, Greg On Mar 20, 2014, at 3:36 PM, Michael Sokolov msoko...@safaribooksonline.com wrote: I'm getting a similar exception when writing documents (on the client side). I can write one document fine, but the second (which is being routed to a different shard) generates the error. It happens every time - definitely not a resource issue or timing problem since this database is completely empty -- I'm just getting started and running some tests, so there must be some kind of setup problem. But it's difficult to diagnose (for me, anyway)! I'd appreciate any insight, hints, guesses, etc. since I'm stuck. Thanks! One node (the leader?) is reporting Internal Server Error in its log, and another node (presumably the shard where the document is being directed) bombs out like this: ERROR - 2014-03-20 15:56:53.022; org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: ERROR adding document SolrInputDocument( ... long dump of document fields ) at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:99) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136) at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121) at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190) at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173) at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106) at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721) ... Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366) at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240) at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:119) at
Re: Rounding errors with SOLR score
I will send the debugQuery. They are exactly the same.
On Fri, Mar 21, 2014 at 2:59 AM, Raymond Wiker rwi...@gmail.com wrote:
Are you sure that SOLR is rounding incorrectly, and not simply differently
from what you expect? I was surprised myself at some of the rounding
behaviour I saw with SOLR, but according to
http://en.wikipedia.org/wiki/Rounding , the results were valid (just not
the round-up-from-half that I naively expected).
On Fri, Mar 21, 2014 at 3:27 AM, William Bell billnb...@gmail.com wrote:
When doing complex boosting/bq we are getting rounding errors on the
score.
To get the score to be consistent I needed to use rint on sort:
sort=rint(product(sum($p_score,$s_score,$q_score),100)) desc,s_query asc
str name=p_scorerecip(priority,1,.5,.01)/str
str name=s_scoreproduct(recip(synonym_rank,1,1,.01),17)/str
str name=q_score
query({!dismax qf=user_query_edge^1 user_query^0.5 user_query_fuzzy
v=$q1})
/str
The issue is in the qf area.
{s_query: Ear Irrigation,score: 10.331313},{s_query: Ear
Piercing,
score: 10.331314},{s_query: Ear Pinning,score: 10.331313},
--
Bill Bell
billnb...@gmail.com
cell 720-256-8076
--
Bill Bell
billnb...@gmail.com
cell 720-256-8076
Re: Best approach to handle large volume of documents with constantly high incoming rate?
Jack, thanks for your reply. Sorry for the confusion about 4 nodes. What I meant was to use 4 nodes to do some POC, mainly focusing on handling the high incoming rate in a few days instead of storing data over one year. You estimated the required nodes (6,308) and storage (322TB) based on the incoming rate and doc size. I have a few questions regarding to them: 1) Is 100 million docs/node some general capacity guideline for a Solr node? 2) Assuming we can provide 6,308 nodes, can Solr Cloud really scale to that level? I found you indicated some common sense limits of Solr Cluster size of 64 nodes in the following mail thread http://find.searchhub.org/document/d823643e65fe2015#84f0c89df2426990 3) If 64 nodes are something we know Solr Cloud can scale up to, then does it mean I can only be sure that 1% of the mentioned workload can be handle by Solr Cloud? (64 is about 1% of 6,308 nodes) 4) The above mentioned Solr Limitations mail thread did mention some cluster with 512 nodes but not really verified whether it worked or not; assuming it worked, it just means we may be able to handle a little less than 10% of the desired workload. 5) Given above simple deduction, it seems 2K docs/sec (10% of the mentioned incoming rate) is the practical limitation of Solr Cloud we can guess for our use case? 6) If the incoming rate is controlled to be around 1k or 2k docs/sec and we want to use Solr Cluster with 64 nodes (or more if it still works), what kind of collection/shard/core structure should be? I am more looking for architectural advice regarding to Solr Cloud structure to handle high incoming rate of relatively small docs. Regards. Shushuai On Saturday, March 22, 2014 2:17 PM, Jack Krupansky j...@basetechnology.com wrote: 20K docs/sec = 20,000 * 60 * 60 * 24 = 1,728,000,000 = 1.7 billion docs/day * 365 = 630,720,000,000 = 631 billion docs/yr At 100 million docs/node = 6,308 nodes! And you think you can do it with 4 nodes? Oh, and that's before replication! 0.5K/doc * 631 billion docs = 322 TB. -- Jack Krupansky -Original Message- From: shushuai zhu Sent: Saturday, March 22, 2014 11:32 AM To: solr-user@lucene.apache.org Subject: Re: Best approach to handle large volume of documents with constantly high incoming rate? Any thoughts? Can Solr Cloud support such use case with acceptable performance? On Thursday, March 20, 2014 7:51 PM, shushuai zhu ss...@yahoo.com wrote: Hi, I am looking for some advice to handle large volume of documents with a very high incoming rate. The size of each document is about 0.5 KB and the incoming rate could be more than 20K per second and we want to store about one year's documents in Solr for near real=time searching. The goal is to achieve acceptable indexing and querying performance. We will use techniques like soft commit, dedicated indexing servers, etc. My main question is about how to structure the collection/shard/core to achieve the goals. Since the incoming rate is very high, we do not want the incoming documents to affect the existing older indexes. Some thoughts are to create a latest index to hold the incoming documents (say latest half hour's data, about 36M docs) so queries on older data could be faster since the old indexes are not affected. There seem three ways to grow the time dimension by adding/splitting/creating a new object listed below every half hour: collection shard core Which is the best way to grow the time dimension? Any limitation in that direction? Or there is some better approach? As an example, I am thinking about having 4 nodes with the following configuration to setup a Solr Cloud: Memory: 128 GB Storage: 4 TB How to set the collection/shard/core to deal with the use case? Thanks in advance. Shushuai
Re: Best approach to handle large volume of documents with constantly high incoming rate?
Well, the commonsense limits Jack is referring to in that post are more (IMO) scales you should count on having to do some _serious_ prototyping/configuring/etc. As you scale out, you'll run into edge cases that aren't the common variety, aren't reliably tested every night, etc. I mean how would you set up a test bed that had 1,000 nodes? Sure, it can be done, but nobody's volunteered yet to provide the Apache Solr project that much hardware. I suspect that it would make Uwe's week if someone did though. In the practical limit vein, one example: You'll run up against the laggard problem. Let's assume that you successfully put up 2,000 nodes, for simplicity's sake, no replicas, just leaders and they all stay up all the time. To successfully do a search, you need to send out a request to all 2,000 nodes. The chance that one of them is slow for _any_ reason (GC, high CPU load, it's just tired) increases the more nodes you have. And since you have to wait until the slowest node responds, your query rate will suffer correspondingly. I've seen 4 node clusters handle 5,000 docs/sec update rate FWIW. YMMV of course. However, you say ...dedicated indexing servers There's no such thing in SolrCloud. Every document gets sent to every member of the slice it belongs to. How else could NRT be supported? When I saw that comment I wondered how well you understand SolrCloud. I flat guarantee you'll understand SolrCloud really, really well if yo try to scale as you indicate :). There'll be a whole bunch of learning experiences along the way, some will be painful. I guarantee that too. Responding to your points 1) Yes, no, and maybe. For relatively small docs on relatively modern hardware, it's a good place to start. Then you have to push it until it falls over to determine your _real_ rates. See: http://searchhub.org/dev/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/ 2) Nobody knows. There's no theoretical reason why SolrCloud shouldn't; no a-priori hard limits. I _strongly_ suspect you'll be on the bleeding edge of size, though. Expect some things to be learning experiences. 3) No, it doesn't mean that at all. 64 is an arbitrary number that means, IMO, here there be dragons. As you start to scale out beyond this you'll run into pesky issues I expect. Your network won't be as reliable as you think. You'll find one of your VMs (which I expect you'll be running on) has some glitches. Someone loaded a very CPU intensive program on three of your machines and your Solrs on those machines is being starved. Etc. 4) I've personally seen 1,000 node clusters. You ought to see the very cool. SolrCloud admin graph I recently saw... But I expect you'll actually be in for some kind of divide-and-conquer strategy whereby you have a bunch of clusters that are significantly smaller. You could, for instance, determine that the use-case you support is searching across small ranges, say a week at a time and have 52 clusters of 128 machines or so. You could have 365 clusters of 20 machines. It all depends on how the index will be used. 5) Not at all. See above, I've seen 5K/sec on 4 nodes, also supporting simultaneous searching. 6) N/A I really can't give you advice. You haven't, for instance, said anything about searches. What kind of SLA are you aiming for? What kind of queries? Faceting? Grouping? I can change the memory footprint of Solr by firing off really ugly queries. In essence, you absolutely must prototype, see the link above. And do a lot of homework defining how you will search the corpus. Otherwise you're guessing. But at this kind of scale, expect to do something other than throw all the docs at a 64 node cluster and expect it to just work. It'll be a lot of work. On Sat, Mar 22, 2014 at 6:48 PM, shushuai zhu ss...@yahoo.com wrote: Jack, thanks for your reply. Sorry for the confusion about 4 nodes. What I meant was to use 4 nodes to do some POC, mainly focusing on handling the high incoming rate in a few days instead of storing data over one year. You estimated the required nodes (6,308) and storage (322TB) based on the incoming rate and doc size. I have a few questions regarding to them: 1) Is 100 million docs/node some general capacity guideline for a Solr node? 2) Assuming we can provide 6,308 nodes, can Solr Cloud really scale to that level? I found you indicated some common sense limits of Solr Cluster size of 64 nodes in the following mail thread http://find.searchhub.org/document/d823643e65fe2015#84f0c89df2426990 3) If 64 nodes are something we know Solr Cloud can scale up to, then does it mean I can only be sure that 1% of the mentioned workload can be handle by Solr Cloud? (64 is about 1% of 6,308 nodes) 4) The above mentioned Solr Limitations mail thread did mention some cluster with 512 nodes but not really verified whether it worked or not; assuming it worked, it just means we may be able to handle a little less than 10% of the desired
Re: Best approach to handle large volume of documents with constantly high incoming rate?
Erick, Thanks a lot for the detailed answers. They are very helpful and I do get some idea from them. As per our searches, we will mainly do term and field (AND/OR) searches, histogram, and faceting. Generally the queries are bound by time (e.g, last hour, last day, last week, or even last month). The queries are not complicated but the challenge is to support near real-time queries with high incoming rate. I have to admit that I just started looking into Solr although I used ElasticSearch for a little while. By default, ElasticSearch creates daily indexes to scale in time dimension and that is the reason I asked if I could customize Solr to do some similar thing to scale out in time dimension. I did notice the following slides talking about scaling time via multiple collections: http://www.slideshare.net/sematext/solr-for-indexing-and-searching-logs but I found more discussions about the limited number of collections Solr can support (not more than 1000 mainly due to znode 1 MB limit from zookeeper?). So, I feel it might be better to scale time via multiple shards or cores (Solr has lotsOfCores feature). Will appreciate very much if more architectural advice could be given for this use case. Regards. Shushuai On Saturday, March 22, 2014 11:10 PM, Erick Erickson erickerick...@gmail.com wrote: Well, the commonsense limits Jack is referring to in that post are more (IMO) scales you should count on having to do some _serious_ prototyping/configuring/etc. As you scale out, you'll run into edge cases that aren't the common variety, aren't reliably tested every night, etc. I mean how would you set up a test bed that had 1,000 nodes? Sure, it can be done, but nobody's volunteered yet to provide the Apache Solr project that much hardware. I suspect that it would make Uwe's week if someone did though. In the practical limit vein, one example: You'll run up against the laggard problem. Let's assume that you successfully put up 2,000 nodes, for simplicity's sake, no replicas, just leaders and they all stay up all the time. To successfully do a search, you need to send out a request to all 2,000 nodes. The chance that one of them is slow for _any_ reason (GC, high CPU load, it's just tired) increases the more nodes you have. And since you have to wait until the slowest node responds, your query rate will suffer correspondingly. I've seen 4 node clusters handle 5,000 docs/sec update rate FWIW. YMMV of course. However, you say ...dedicated indexing servers There's no such thing in SolrCloud. Every document gets sent to every member of the slice it belongs to. How else could NRT be supported? When I saw that comment I wondered how well you understand SolrCloud. I flat guarantee you'll understand SolrCloud really, really well if yo try to scale as you indicate :). There'll be a whole bunch of learning experiences along the way, some will be painful. I guarantee that too. Responding to your points 1) Yes, no, and maybe. For relatively small docs on relatively modern hardware, it's a good place to start. Then you have to push it until it falls over to determine your _real_ rates. See: http://searchhub.org/dev/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/ 2) Nobody knows. There's no theoretical reason why SolrCloud shouldn't; no a-priori hard limits. I _strongly_ suspect you'll be on the bleeding edge of size, though. Expect some things to be learning experiences. 3) No, it doesn't mean that at all. 64 is an arbitrary number that means, IMO, here there be dragons. As you start to scale out beyond this you'll run into pesky issues I expect. Your network won't be as reliable as you think. You'll find one of your VMs (which I expect you'll be running on) has some glitches. Someone loaded a very CPU intensive program on three of your machines and your Solrs on those machines is being starved. Etc. 4) I've personally seen 1,000 node clusters. You ought to see the very cool. SolrCloud admin graph I recently saw... But I expect you'll actually be in for some kind of divide-and-conquer strategy whereby you have a bunch of clusters that are significantly smaller. You could, for instance, determine that the use-case you support is searching across small ranges, say a week at a time and have 52 clusters of 128 machines or so. You could have 365 clusters of 20 machines. It all depends on how the index will be used. 5) Not at all. See above, I've seen 5K/sec on 4 nodes, also supporting simultaneous searching. 6) N/A I really can't give you advice. You haven't, for instance, said anything about searches. What kind of SLA are you aiming for? What kind of queries? Faceting? Grouping? I can change the memory footprint of Solr by firing off really ugly queries. In essence, you absolutely must prototype, see the link above. And do a lot of homework defining how you will search the corpus. Otherwise you're guessing. But at this kind of scale, expect to do
Re: Best approach to handle large volume of documents with constantly high incoming rate?
I defer to Erick on on this level of detail and experience. Let's continue the discussion - some of it will be a matter of how to configure and tune Solr, how to select, configure, and tune hardware, the need for further Lucene/Solr improvements, and how much further we have to go to get to the next level with Big Data. I mean, 20K events/sec is not necessarily beyond the realm of reality these days with sensor data (20K/sec = 1 event every 50 microseconds) -- Jack Krupansky -Original Message- From: Erick Erickson Sent: Saturday, March 22, 2014 11:02 PM To: solr-user@lucene.apache.org ; shushuai zhu Subject: Re: Best approach to handle large volume of documents with constantly high incoming rate? Well, the commonsense limits Jack is referring to in that post are more (IMO) scales you should count on having to do some _serious_ prototyping/configuring/etc. As you scale out, you'll run into edge cases that aren't the common variety, aren't reliably tested every night, etc. I mean how would you set up a test bed that had 1,000 nodes? Sure, it can be done, but nobody's volunteered yet to provide the Apache Solr project that much hardware. I suspect that it would make Uwe's week if someone did though. In the practical limit vein, one example: You'll run up against the laggard problem. Let's assume that you successfully put up 2,000 nodes, for simplicity's sake, no replicas, just leaders and they all stay up all the time. To successfully do a search, you need to send out a request to all 2,000 nodes. The chance that one of them is slow for _any_ reason (GC, high CPU load, it's just tired) increases the more nodes you have. And since you have to wait until the slowest node responds, your query rate will suffer correspondingly. I've seen 4 node clusters handle 5,000 docs/sec update rate FWIW. YMMV of course. However, you say ...dedicated indexing servers There's no such thing in SolrCloud. Every document gets sent to every member of the slice it belongs to. How else could NRT be supported? When I saw that comment I wondered how well you understand SolrCloud. I flat guarantee you'll understand SolrCloud really, really well if yo try to scale as you indicate :). There'll be a whole bunch of learning experiences along the way, some will be painful. I guarantee that too. Responding to your points 1) Yes, no, and maybe. For relatively small docs on relatively modern hardware, it's a good place to start. Then you have to push it until it falls over to determine your _real_ rates. See: http://searchhub.org/dev/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/ 2) Nobody knows. There's no theoretical reason why SolrCloud shouldn't; no a-priori hard limits. I _strongly_ suspect you'll be on the bleeding edge of size, though. Expect some things to be learning experiences. 3) No, it doesn't mean that at all. 64 is an arbitrary number that means, IMO, here there be dragons. As you start to scale out beyond this you'll run into pesky issues I expect. Your network won't be as reliable as you think. You'll find one of your VMs (which I expect you'll be running on) has some glitches. Someone loaded a very CPU intensive program on three of your machines and your Solrs on those machines is being starved. Etc. 4) I've personally seen 1,000 node clusters. You ought to see the very cool. SolrCloud admin graph I recently saw... But I expect you'll actually be in for some kind of divide-and-conquer strategy whereby you have a bunch of clusters that are significantly smaller. You could, for instance, determine that the use-case you support is searching across small ranges, say a week at a time and have 52 clusters of 128 machines or so. You could have 365 clusters of 20 machines. It all depends on how the index will be used. 5) Not at all. See above, I've seen 5K/sec on 4 nodes, also supporting simultaneous searching. 6) N/A I really can't give you advice. You haven't, for instance, said anything about searches. What kind of SLA are you aiming for? What kind of queries? Faceting? Grouping? I can change the memory footprint of Solr by firing off really ugly queries. In essence, you absolutely must prototype, see the link above. And do a lot of homework defining how you will search the corpus. Otherwise you're guessing. But at this kind of scale, expect to do something other than throw all the docs at a 64 node cluster and expect it to just work. It'll be a lot of work. On Sat, Mar 22, 2014 at 6:48 PM, shushuai zhu ss...@yahoo.com wrote: Jack, thanks for your reply. Sorry for the confusion about 4 nodes. What I meant was to use 4 nodes to do some POC, mainly focusing on handling the high incoming rate in a few days instead of storing data over one year. You estimated the required nodes (6,308) and storage (322TB) based on the incoming rate and doc size. I have a few questions regarding to them: 1) Is 100 million docs/node some general capacity guideline for a
