Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
Just to give folks an update, we trashed the server having issues and cloned/rebuild a VM from a sane server and it seems to be running good for the past 3 days without any issues. We intend to monitor it over the weekend. If its still stable on Monday, I would blame the issues it on the server configuration. :-) Thanks Ravi Kiran Bhaskar On Tue, May 15, 2012 at 2:57 PM, Ravi Solr ravis...@gmail.com wrote: I have already triple cross-checked that all my clients are using same version as the server which is 3.6 Thanks Ravi Kiran On Tue, May 15, 2012 at 2:09 PM, Ramesh K Balasubramanian beeyar...@yahoo.com wrote: I have seen similar errors before when the solr version and solrj version in the client don't match. Best Regards, Ramesh
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
Hello, Unfortunately it seems like I spoke too early. Today morning I received the same error again even after disabling the iptables. The weird thing is only one out of 6 or 7 queries fails as evidenced in the stack traces below. The query below the stack trace gave a 'status=500' subsequent queries look fine [#|2012-05-15T08:12:38.703-0400|SEVERE|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-8;_RequestID=9f54ea89-357a-4c1b-87a1-fbaacc9fd0ee;|org.apache.solr.common.SolrException at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:275) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111) Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99) at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:469) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:249) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:129) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:103) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
I have seen similar errors before when the solr version and solrj version in the client don't match. Best Regards, Ramesh
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
I have already triple cross-checked that all my clients are using same version as the server which is 3.6 Thanks Ravi Kiran On Tue, May 15, 2012 at 2:09 PM, Ramesh K Balasubramanian beeyar...@yahoo.com wrote: I have seen similar errors before when the solr version and solrj version in the client don't match. Best Regards, Ramesh
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
Guys, just to give you an update, we think we might have found the issue. iptables was enabled on one query server and disabled on the other. The server where iptables is enabled is the one having issues, we disabled the iptables today to test out the theory that the iptables might be causing this issue of null/empty response. If the server holds up during the weekend then we have the culprit :-) Thanks to all of you who helped me out. Stay tuned. Ravi Kiran On Fri, May 11, 2012 at 1:23 AM, Shawn Heisey s...@elyograg.org wrote: On 5/10/2012 4:17 PM, Ravi Solr wrote: Thanks for responding Mr. Heisey... I don't see any parsing errors in my log but I see lot of exceptions like the one listed belowonce an exception like this happens weirdness ensues. For example - To check sanity I queried for uniquekey:111 from the solr admin GUI it gave back numFound equal to all docs in that index i.e. its not searching for that uniquekey at all, it blindly matched all docs. However, once you restart the server the same index without any change works perfectly returning only one doc in numFound when you search for uniquekey:111...I tried everything from reindexing, copying index from another sane server, delete entire index and reindex from scratch etc but in vain, it works for roughly 24 hours and then starts throwing the same error no matter what the query is. [#|2012-05-10T13:27:14.071-0400|SEVERE|sun-appserver2.1.1|xxx.xxx.xxx.xxx|_ThreadID=21;_ThreadName=httpSSLWorkerThread-9001-6;_RequestID=d44462e7-576b-4391-a499-c65da33e3293;|Error searching data for section Local org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) at xxx.xxx.xxx.xxx(FeedController.java:621) at xxx.xxx.xxx.xxx(FeedController.java:402) This is still saying solrj. Unless I am completely misunderstanding the way things work, which I will freely admit is possible, this is the client code. Do you have anything in the log files from Solr (the server)? I don't have a lot of experience with Tomcat, because I run my Solr under jetty as included in the example. It looks like the client is running under Tomcat, though I suppose you might be running Solr under a different container. Thanks, Shawn
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
Yeah, 9 times out of 10, this error is a 404 - which wouldn't be logged anywhere. On May 11, 2012, at 6:12 PM, Ravi Solr wrote: Guys, just to give you an update, we think we might have found the issue. iptables was enabled on one query server and disabled on the other. The server where iptables is enabled is the one having issues, we disabled the iptables today to test out the theory that the iptables might be causing this issue of null/empty response. If the server holds up during the weekend then we have the culprit :-) Thanks to all of you who helped me out. Stay tuned. Ravi Kiran On Fri, May 11, 2012 at 1:23 AM, Shawn Heisey s...@elyograg.org wrote: On 5/10/2012 4:17 PM, Ravi Solr wrote: Thanks for responding Mr. Heisey... I don't see any parsing errors in my log but I see lot of exceptions like the one listed belowonce an exception like this happens weirdness ensues. For example - To check sanity I queried for uniquekey:111 from the solr admin GUI it gave back numFound equal to all docs in that index i.e. its not searching for that uniquekey at all, it blindly matched all docs. However, once you restart the server the same index without any change works perfectly returning only one doc in numFound when you search for uniquekey:111...I tried everything from reindexing, copying index from another sane server, delete entire index and reindex from scratch etc but in vain, it works for roughly 24 hours and then starts throwing the same error no matter what the query is. [#|2012-05-10T13:27:14.071-0400|SEVERE|sun-appserver2.1.1|xxx.xxx.xxx.xxx|_ThreadID=21;_ThreadName=httpSSLWorkerThread-9001-6;_RequestID=d44462e7-576b-4391-a499-c65da33e3293;|Error searching data for section Local org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) at xxx.xxx.xxx.xxx(FeedController.java:621) at xxx.xxx.xxx.xxx(FeedController.java:402) This is still saying solrj. Unless I am completely misunderstanding the way things work, which I will freely admit is possible, this is the client code. Do you have anything in the log files from Solr (the server)? I don't have a lot of experience with Tomcat, because I run my Solr under jetty as included in the example. It looks like the client is running under Tomcat, though I suppose you might be running Solr under a different container. Thanks, Shawn - Mark Miller lucidimagination.com
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
I clean the entire index and re-indexed it with SOLRJ 3.6. Still I get the same error every single day. How can I see if the container returned partial/nonconforming response since it may be hidden by solrj ? Thanks Ravi Kiran Bhaskar On Mon, May 7, 2012 at 2:16 PM, Ravi Solr ravis...@gmail.com wrote: Hello Mr. Miller and Mr. Erickson, Found yet another inconsistency on the query server that might be causing this issue. Today morning also I got a similar error as shown in stacktrace below. So I tried querying for that d101dd3a-979a-11e1-927c-291130c98dff which is our unique key in the schema. On the server having issue it returned more than 10 docs with numFound=1051273 and on all other sane servers it returned only 1 doc with numFound=1. This is really weird, as, we copied the entire index from a sane server onto the server having issues now just 2 days ago. Do you have any idea why this would happen ? [#|2012-05-07T12:58:54.055-0400|SEVERE|sun-appserver2.1.1|com.wpost.ipad.feeds.FeedController|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-3;_RequestID=4203e3e5-c39d-4df7-a32a-600d0169c81f;|Error searching for thumbnails for d101dd3a-979a-11e1-927c-291130c98dff org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) at xxx.xxx.xxx.xxx.populateThumbnails(FeedController.java:1184) at xxx.xxx.xxx.xxx..findNewsBySection(FeedController.java:509) at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source) .. ... .. Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99) at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89) ... 43 more |#] Ravi Kiran Bhaskar Principal Software Engineer Washington Post Digital 1150 15th Street NW, Washington, DC 20071 On Mon, May 7, 2012 at 9:36 AM, Mark Miller markrmil...@gmail.com wrote: Normally this specific error is caused by a non success http error page and response is returned. The response parser tries to parse HTML as javabin. Sent from my iPhone On May 7, 2012, at 7:37 AM, Erick Erickson erickerick...@gmail.com wrote: Well, I'm guessing that the version of Solr (and perhaps there are classpath issues in here?) are different, somehow, on the machine slave that is showing the error. It's also possible that your config files have a different LUCENE_VERSION in them, although I don't think this should really create the errors you're reporting. The thing that leads me in this direction is your statement that things are fine for a while and then go bad later. If replication happens just before you get the index version error, that would point a finger at something like different Solr versions. If there is no replication before this error, then this probably isn't the problem and we'll have to look elsewhere... But this is all guesswork, just like every bug... things are only obvious after you find the problem! Best Erick On Sun, May 6, 2012 at 11:08 AM, Ravi Solr ravis...@gmail.com wrote: Thank you very much for responding Mr.Erickson. You may be right on old version index, I will reindex. However we have a 2 separate/disjoint master-slave setup...only one query node/slave has this issue. if it was really incompatible indexes why isnt the other query server also throwing errors? that's what is throwing my debugging thought process off. Thanks Ravi Kiran Bhaskar Principal Software Engineer Washington Post Digital 1150 15th Street NW, Washington, DC 20071 On Sat, May 5, 2012 at 12:53 PM, Erick Erickson erickerick...@gmail.com wrote: The first thing I'd check is if, in the log, there is a replication happening immediately prior to the error. I confess I'm not entirely up on the version thing, but is it possible you're replicating an index that is built with some other version of Solr? That would at least explain your statement that it runs OK, but then fails sometime later. Best Erick On Fri, May 4, 2012 at 1:50 PM, Ravi Solr ravis...@gmail.com wrote: Hello, We Recently we migrated our SOLR 3.6 server OS from Solaris to CentOS and from then on we started seeing Invalid version (expected 2, but 60) errors on one of the query servers (oddly one other query server seems fine). If we restart the server having issue everything will be alright, but the next
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
On 5/10/2012 12:27 PM, Ravi Solr wrote: I clean the entire index and re-indexed it with SOLRJ 3.6. Still I get the same error every single day. How can I see if the container returned partial/nonconforming response since it may be hidden by solrj ? If the server is sending a non-javabin error response that SolrJ doesn't parse, the logs from the container that runs Solr will normally give you useful information. Here's the first part of something logged on mine for an invalid query - the query sent in only had one double quote. The log actually contains the full Java stacktrace, I just included the first little bit: May 10, 2012 2:13:17 PM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: org.apache.lucene.queryParser.ParseException: Cannot parse '( (trader joe's))': Lexical error at line 1, column 20. Encountered: EOF after : \trader joe\'s)) Thanks, Shawn
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
Thanks for responding Mr. Heisey... I don't see any parsing errors in my log but I see lot of exceptions like the one listed belowonce an exception like this happens weirdness ensues. For example - To check sanity I queried for uniquekey:111 from the solr admin GUI it gave back numFound equal to all docs in that index i.e. its not searching for that uniquekey at all, it blindly matched all docs. However, once you restart the server the same index without any change works perfectly returning only one doc in numFound when you search for uniquekey:111...I tried everything from reindexing, copying index from another sane server, delete entire index and reindex from scratch etc but in vain, it works for roughly 24 hours and then starts throwing the same error no matter what the query is. [#|2012-05-10T13:27:14.071-0400|SEVERE|sun-appserver2.1.1|xxx.xxx.xxx.xxx|_ThreadID=21;_ThreadName=httpSSLWorkerThread-9001-6;_RequestID=d44462e7-576b-4391-a499-c65da33e3293;|Error searching data for section Local org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) at xxx.xxx.xxx.xxx(FeedController.java:621) at xxx.xxx.xxx.xxx(FeedController.java:402) at sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549) at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221) at
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
On 5/10/2012 4:17 PM, Ravi Solr wrote: Thanks for responding Mr. Heisey... I don't see any parsing errors in my log but I see lot of exceptions like the one listed belowonce an exception like this happens weirdness ensues. For example - To check sanity I queried for uniquekey:111 from the solr admin GUI it gave back numFound equal to all docs in that index i.e. its not searching for that uniquekey at all, it blindly matched all docs. However, once you restart the server the same index without any change works perfectly returning only one doc in numFound when you search for uniquekey:111...I tried everything from reindexing, copying index from another sane server, delete entire index and reindex from scratch etc but in vain, it works for roughly 24 hours and then starts throwing the same error no matter what the query is. [#|2012-05-10T13:27:14.071-0400|SEVERE|sun-appserver2.1.1|xxx.xxx.xxx.xxx|_ThreadID=21;_ThreadName=httpSSLWorkerThread-9001-6;_RequestID=d44462e7-576b-4391-a499-c65da33e3293;|Error searching data for section Local org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) at xxx.xxx.xxx.xxx(FeedController.java:621) at xxx.xxx.xxx.xxx(FeedController.java:402) This is still saying solrj. Unless I am completely misunderstanding the way things work, which I will freely admit is possible, this is the client code. Do you have anything in the log files from Solr (the server)? I don't have a lot of experience with Tomcat, because I run my Solr under jetty as included in the example. It looks like the client is running under Tomcat, though I suppose you might be running Solr under a different container. Thanks, Shawn
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
Well, I'm guessing that the version of Solr (and perhaps there are classpath issues in here?) are different, somehow, on the machine slave that is showing the error. It's also possible that your config files have a different LUCENE_VERSION in them, although I don't think this should really create the errors you're reporting. The thing that leads me in this direction is your statement that things are fine for a while and then go bad later. If replication happens just before you get the index version error, that would point a finger at something like different Solr versions. If there is no replication before this error, then this probably isn't the problem and we'll have to look elsewhere... But this is all guesswork, just like every bug... things are only obvious after you find the problem! Best Erick On Sun, May 6, 2012 at 11:08 AM, Ravi Solr ravis...@gmail.com wrote: Thank you very much for responding Mr.Erickson. You may be right on old version index, I will reindex. However we have a 2 separate/disjoint master-slave setup...only one query node/slave has this issue. if it was really incompatible indexes why isnt the other query server also throwing errors? that's what is throwing my debugging thought process off. Thanks Ravi Kiran Bhaskar Principal Software Engineer Washington Post Digital 1150 15th Street NW, Washington, DC 20071 On Sat, May 5, 2012 at 12:53 PM, Erick Erickson erickerick...@gmail.com wrote: The first thing I'd check is if, in the log, there is a replication happening immediately prior to the error. I confess I'm not entirely up on the version thing, but is it possible you're replicating an index that is built with some other version of Solr? That would at least explain your statement that it runs OK, but then fails sometime later. Best Erick On Fri, May 4, 2012 at 1:50 PM, Ravi Solr ravis...@gmail.com wrote: Hello, We Recently we migrated our SOLR 3.6 server OS from Solaris to CentOS and from then on we started seeing Invalid version (expected 2, but 60) errors on one of the query servers (oddly one other query server seems fine). If we restart the server having issue everything will be alright, but the next day in the morning again we get the same exception. I made sure that all the client applications are using SOLR 3.6 version. The Glassfish on which all the applications and SOLR are deployed use Java 1.6.0_29. The only difference I could see 1. The process indexing to the server having issues is using java1.6.0_31 2. The process indexing to the server that DOES NOT have issues is using java1.6.0_29 Could the Java minor version being greater than the SOLR instance be the cause of this issue ??? Can anybody please help me debug this a bit more ? what else can I look at to understand the underlying problem. The stack trace is given below [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx...|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5; org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) at com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743) at com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347) at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549) at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315) at
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
Normally this specific error is caused by a non success http error page and response is returned. The response parser tries to parse HTML as javabin. Sent from my iPhone On May 7, 2012, at 7:37 AM, Erick Erickson erickerick...@gmail.com wrote: Well, I'm guessing that the version of Solr (and perhaps there are classpath issues in here?) are different, somehow, on the machine slave that is showing the error. It's also possible that your config files have a different LUCENE_VERSION in them, although I don't think this should really create the errors you're reporting. The thing that leads me in this direction is your statement that things are fine for a while and then go bad later. If replication happens just before you get the index version error, that would point a finger at something like different Solr versions. If there is no replication before this error, then this probably isn't the problem and we'll have to look elsewhere... But this is all guesswork, just like every bug... things are only obvious after you find the problem! Best Erick On Sun, May 6, 2012 at 11:08 AM, Ravi Solr ravis...@gmail.com wrote: Thank you very much for responding Mr.Erickson. You may be right on old version index, I will reindex. However we have a 2 separate/disjoint master-slave setup...only one query node/slave has this issue. if it was really incompatible indexes why isnt the other query server also throwing errors? that's what is throwing my debugging thought process off. Thanks Ravi Kiran Bhaskar Principal Software Engineer Washington Post Digital 1150 15th Street NW, Washington, DC 20071 On Sat, May 5, 2012 at 12:53 PM, Erick Erickson erickerick...@gmail.com wrote: The first thing I'd check is if, in the log, there is a replication happening immediately prior to the error. I confess I'm not entirely up on the version thing, but is it possible you're replicating an index that is built with some other version of Solr? That would at least explain your statement that it runs OK, but then fails sometime later. Best Erick On Fri, May 4, 2012 at 1:50 PM, Ravi Solr ravis...@gmail.com wrote: Hello, We Recently we migrated our SOLR 3.6 server OS from Solaris to CentOS and from then on we started seeing Invalid version (expected 2, but 60) errors on one of the query servers (oddly one other query server seems fine). If we restart the server having issue everything will be alright, but the next day in the morning again we get the same exception. I made sure that all the client applications are using SOLR 3.6 version. The Glassfish on which all the applications and SOLR are deployed use Java 1.6.0_29. The only difference I could see 1. The process indexing to the server having issues is using java1.6.0_31 2. The process indexing to the server that DOES NOT have issues is using java1.6.0_29 Could the Java minor version being greater than the SOLR instance be the cause of this issue ??? Can anybody please help me debug this a bit more ? what else can I look at to understand the underlying problem. The stack trace is given below [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx...|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5; org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) at com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743) at com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347) at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549) at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
Hello Mr. Miller and Mr. Erickson, Found yet another inconsistency on the query server that might be causing this issue. Today morning also I got a similar error as shown in stacktrace below. So I tried querying for that d101dd3a-979a-11e1-927c-291130c98dff which is our unique key in the schema. On the server having issue it returned more than 10 docs with numFound=1051273 and on all other sane servers it returned only 1 doc with numFound=1. This is really weird, as, we copied the entire index from a sane server onto the server having issues now just 2 days ago. Do you have any idea why this would happen ? [#|2012-05-07T12:58:54.055-0400|SEVERE|sun-appserver2.1.1|com.wpost.ipad.feeds.FeedController|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-3;_RequestID=4203e3e5-c39d-4df7-a32a-600d0169c81f;|Error searching for thumbnails for d101dd3a-979a-11e1-927c-291130c98dff org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) at xxx.xxx.xxx.xxx.populateThumbnails(FeedController.java:1184) at xxx.xxx.xxx.xxx..findNewsBySection(FeedController.java:509) at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source) .. ... .. Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99) at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89) ... 43 more |#] Ravi Kiran Bhaskar Principal Software Engineer Washington Post Digital 1150 15th Street NW, Washington, DC 20071 On Mon, May 7, 2012 at 9:36 AM, Mark Miller markrmil...@gmail.com wrote: Normally this specific error is caused by a non success http error page and response is returned. The response parser tries to parse HTML as javabin. Sent from my iPhone On May 7, 2012, at 7:37 AM, Erick Erickson erickerick...@gmail.com wrote: Well, I'm guessing that the version of Solr (and perhaps there are classpath issues in here?) are different, somehow, on the machine slave that is showing the error. It's also possible that your config files have a different LUCENE_VERSION in them, although I don't think this should really create the errors you're reporting. The thing that leads me in this direction is your statement that things are fine for a while and then go bad later. If replication happens just before you get the index version error, that would point a finger at something like different Solr versions. If there is no replication before this error, then this probably isn't the problem and we'll have to look elsewhere... But this is all guesswork, just like every bug... things are only obvious after you find the problem! Best Erick On Sun, May 6, 2012 at 11:08 AM, Ravi Solr ravis...@gmail.com wrote: Thank you very much for responding Mr.Erickson. You may be right on old version index, I will reindex. However we have a 2 separate/disjoint master-slave setup...only one query node/slave has this issue. if it was really incompatible indexes why isnt the other query server also throwing errors? that's what is throwing my debugging thought process off. Thanks Ravi Kiran Bhaskar Principal Software Engineer Washington Post Digital 1150 15th Street NW, Washington, DC 20071 On Sat, May 5, 2012 at 12:53 PM, Erick Erickson erickerick...@gmail.com wrote: The first thing I'd check is if, in the log, there is a replication happening immediately prior to the error. I confess I'm not entirely up on the version thing, but is it possible you're replicating an index that is built with some other version of Solr? That would at least explain your statement that it runs OK, but then fails sometime later. Best Erick On Fri, May 4, 2012 at 1:50 PM, Ravi Solr ravis...@gmail.com wrote: Hello, We Recently we migrated our SOLR 3.6 server OS from Solaris to CentOS and from then on we started seeing Invalid version (expected 2, but 60) errors on one of the query servers (oddly one other query server seems fine). If we restart the server having issue everything will be alright, but the next day in the morning again we get the same exception. I made sure that all the client applications are using SOLR 3.6 version. The Glassfish on which all the applications and SOLR are deployed use Java 1.6.0_29. The only difference I could see 1. The process indexing to the server having issues is using java1.6.0_31 2. The
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
Thank you very much for responding Mr.Erickson. You may be right on old version index, I will reindex. However we have a 2 separate/disjoint master-slave setup...only one query node/slave has this issue. if it was really incompatible indexes why isnt the other query server also throwing errors? that's what is throwing my debugging thought process off. Thanks Ravi Kiran Bhaskar Principal Software Engineer Washington Post Digital 1150 15th Street NW, Washington, DC 20071 On Sat, May 5, 2012 at 12:53 PM, Erick Erickson erickerick...@gmail.com wrote: The first thing I'd check is if, in the log, there is a replication happening immediately prior to the error. I confess I'm not entirely up on the version thing, but is it possible you're replicating an index that is built with some other version of Solr? That would at least explain your statement that it runs OK, but then fails sometime later. Best Erick On Fri, May 4, 2012 at 1:50 PM, Ravi Solr ravis...@gmail.com wrote: Hello, We Recently we migrated our SOLR 3.6 server OS from Solaris to CentOS and from then on we started seeing Invalid version (expected 2, but 60) errors on one of the query servers (oddly one other query server seems fine). If we restart the server having issue everything will be alright, but the next day in the morning again we get the same exception. I made sure that all the client applications are using SOLR 3.6 version. The Glassfish on which all the applications and SOLR are deployed use Java 1.6.0_29. The only difference I could see 1. The process indexing to the server having issues is using java1.6.0_31 2. The process indexing to the server that DOES NOT have issues is using java1.6.0_29 Could the Java minor version being greater than the SOLR instance be the cause of this issue ??? Can anybody please help me debug this a bit more ? what else can I look at to understand the underlying problem. The stack trace is given below [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx...|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5; org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) at com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743) at com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347) at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549) at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587) at
Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!
The first thing I'd check is if, in the log, there is a replication happening immediately prior to the error. I confess I'm not entirely up on the version thing, but is it possible you're replicating an index that is built with some other version of Solr? That would at least explain your statement that it runs OK, but then fails sometime later. Best Erick On Fri, May 4, 2012 at 1:50 PM, Ravi Solr ravis...@gmail.com wrote: Hello, We Recently we migrated our SOLR 3.6 server OS from Solaris to CentOS and from then on we started seeing Invalid version (expected 2, but 60) errors on one of the query servers (oddly one other query server seems fine). If we restart the server having issue everything will be alright, but the next day in the morning again we get the same exception. I made sure that all the client applications are using SOLR 3.6 version. The Glassfish on which all the applications and SOLR are deployed use Java 1.6.0_29. The only difference I could see 1. The process indexing to the server having issues is using java1.6.0_31 2. The process indexing to the server that DOES NOT have issues is using java1.6.0_29 Could the Java minor version being greater than the SOLR instance be the cause of this issue ??? Can anybody please help me debug this a bit more ? what else can I look at to understand the underlying problem. The stack trace is given below [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx...|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5; org.apache.solr.client.solrj.SolrServerException: Error executing query at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) at com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743) at com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347) at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549) at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
