[jira] [Commented] (SOLR-5402) SolrCloud 4.5 bulk add errors in cloud setup

2013-11-19 Thread Mark Miller (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13827109#comment-13827109
 ] 

Mark Miller commented on SOLR-5402:
---

FYI, 4.6 no longer sends updates via XML - so I have not looked into this at 
all yet, but there is some chance the problem has been sidestepped.

 SolrCloud 4.5 bulk add errors in cloud setup
 

 Key: SOLR-5402
 URL: https://issues.apache.org/jira/browse/SOLR-5402
 Project: Solr
  Issue Type: Bug
  Components: SolrCloud
Affects Versions: 4.5, 4.5.1
Reporter: Sai Gadde
 Fix For: 5.0, 4.7


 We use out of the box Solr 4.5.1 no customization done. If we merge documents 
 via SolrJ to a single server it is perfectly working fine.
 But as soon as we add another node to the cloud we are getting following 
 while merging documents. We merge about 500 at a time using SolrJ. These 500 
 documents in total are about few MB (1-3) in size.
 This is the error we are getting on the server (10.10.10.116 - IP is 
 irrelavent just for clarity)where merging is happening. 10.10.10.119 is the 
 new node here. This server gets RemoteSolrException
 shard update error StdNode: 
 http://10.10.10.119:8980/solr/mycore/:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
  Illegal to have multiple roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:425)
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:401)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
 Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
 On the other server 10.10.10.119 we get following error
 org.apache.solr.common.SolrException: Illegal to have multiple roots (start 
 tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176)
   at 
 org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
   at 
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
   at 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
   at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at 
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
   at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
   at 
 org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
   at 
 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at 
 org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal to have multiple 
 roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12369]
   at 
 

[jira] [Commented] (SOLR-5402) SolrCloud 4.5 bulk add errors in cloud setup

2013-11-18 Thread Mark Miller (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13826110#comment-13826110
 ] 

Mark Miller commented on SOLR-5402:
---

Sorry I have not had time to help on this one. It's on my list to look into.

 SolrCloud 4.5 bulk add errors in cloud setup
 

 Key: SOLR-5402
 URL: https://issues.apache.org/jira/browse/SOLR-5402
 Project: Solr
  Issue Type: Bug
  Components: SolrCloud
Affects Versions: 4.5, 4.5.1
Reporter: Sai Gadde
 Fix For: 4.6


 We use out of the box Solr 4.5.1 no customization done. If we merge documents 
 via SolrJ to a single server it is perfectly working fine.
 But as soon as we add another node to the cloud we are getting following 
 while merging documents. We merge about 500 at a time using SolrJ. These 500 
 documents in total are about few MB (1-3) in size.
 This is the error we are getting on the server (10.10.10.116 - IP is 
 irrelavent just for clarity)where merging is happening. 10.10.10.119 is the 
 new node here. This server gets RemoteSolrException
 shard update error StdNode: 
 http://10.10.10.119:8980/solr/mycore/:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
  Illegal to have multiple roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:425)
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:401)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
 Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
 On the other server 10.10.10.119 we get following error
 org.apache.solr.common.SolrException: Illegal to have multiple roots (start 
 tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176)
   at 
 org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
   at 
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
   at 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
   at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at 
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
   at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
   at 
 org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
   at 
 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at 
 org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal to have multiple 
 roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12369]
   at 
 com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:630)
   at 

[jira] [Commented] (SOLR-5402) SolrCloud 4.5 bulk add errors in cloud setup

2013-11-15 Thread Dave Seltzer (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13824158#comment-13824158
 ] 

Dave Seltzer commented on SOLR-5402:


We're having the same issue over here. I was able to insert hundreds of 
documents per add using the REST API and a single collection.

When I re-created my collection as as a cloud instance with NumShards=2, It 
fails whenever I try to insert more than one document per add request.

 SolrCloud 4.5 bulk add errors in cloud setup
 

 Key: SOLR-5402
 URL: https://issues.apache.org/jira/browse/SOLR-5402
 Project: Solr
  Issue Type: Bug
  Components: SolrCloud
Affects Versions: 4.5, 4.5.1
Reporter: Sai Gadde
 Fix For: 4.6


 We use out of the box Solr 4.5.1 no customization done. If we merge documents 
 via SolrJ to a single server it is perfectly working fine.
 But as soon as we add another node to the cloud we are getting following 
 while merging documents. We merge about 500 at a time using SolrJ. These 500 
 documents in total are about few MB (1-3) in size.
 This is the error we are getting on the server (10.10.10.116 - IP is 
 irrelavent just for clarity)where merging is happening. 10.10.10.119 is the 
 new node here. This server gets RemoteSolrException
 shard update error StdNode: 
 http://10.10.10.119:8980/solr/mycore/:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
  Illegal to have multiple roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:425)
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:401)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
 Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
 On the other server 10.10.10.119 we get following error
 org.apache.solr.common.SolrException: Illegal to have multiple roots (start 
 tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176)
   at 
 org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
   at 
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
   at 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
   at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at 
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
   at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
   at 
 org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
   at 
 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at 
 org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal to have 

[jira] [Commented] (SOLR-5402) SolrCloud 4.5 bulk add errors in cloud setup

2013-11-14 Thread Greg Walters (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822728#comment-13822728
 ] 

Greg Walters commented on SOLR-5402:


I've been able to reproduce this issue using curl to add documents but using 
the post.jar provided in the Solr example or using a solrj client I'm unable to 
reproduce this issue having tried batches up to 5000 documents.

 SolrCloud 4.5 bulk add errors in cloud setup
 

 Key: SOLR-5402
 URL: https://issues.apache.org/jira/browse/SOLR-5402
 Project: Solr
  Issue Type: Bug
  Components: SolrCloud
Affects Versions: 4.5, 4.5.1
Reporter: Sai Gadde
 Fix For: 4.6


 We use out of the box Solr 4.5.1 no customization done. If we merge documents 
 via SolrJ to a single server it is perfectly working fine.
 But as soon as we add another node to the cloud we are getting following 
 while merging documents. We merge about 500 at a time using SolrJ. These 500 
 documents in total are about few MB (1-3) in size.
 This is the error we are getting on the server (10.10.10.116 - IP is 
 irrelavent just for clarity)where merging is happening. 10.10.10.119 is the 
 new node here. This server gets RemoteSolrException
 shard update error StdNode: 
 http://10.10.10.119:8980/solr/mycore/:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
  Illegal to have multiple roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:425)
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:401)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
 Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
 On the other server 10.10.10.119 we get following error
 org.apache.solr.common.SolrException: Illegal to have multiple roots (start 
 tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176)
   at 
 org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
   at 
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
   at 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
   at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at 
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
   at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
   at 
 org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
   at 
 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at 
 org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal to have multiple 
 roots (start tag in epilog?).
  at [row,col 

[jira] [Commented] (SOLR-5402) SolrCloud 4.5 bulk add errors in cloud setup

2013-10-31 Thread Michael Tracey (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810551#comment-13810551
 ] 

Michael Tracey commented on SOLR-5402:
--

I can confirm the same thing, it errors with a single document.  That's the 
last test I did before rolling my infrastructure back to 4.4.  I can index 
1000's of documents at a time without issue to a single server, but when 
SolrCloud 4.5.1 tries to sync two nodes, it fails (same errors as above).

 SolrCloud 4.5 bulk add errors in cloud setup
 

 Key: SOLR-5402
 URL: https://issues.apache.org/jira/browse/SOLR-5402
 Project: Solr
  Issue Type: Bug
  Components: SolrCloud
Affects Versions: 4.5, 4.5.1
Reporter: Sai Gadde
 Fix For: 4.6


 We use out of the box Solr 4.5.1 no customization done. If we merge documents 
 via SolrJ to a single server it is perfectly working fine.
 But as soon as we add another node to the cloud we are getting following 
 while merging documents. We merge about 500 at a time using SolrJ. These 500 
 documents in total are about few MB (1-3) in size.
 This is the error we are getting on the server (10.10.10.116 - IP is 
 irrelavent just for clarity)where merging is happening. 10.10.10.119 is the 
 new node here. This server gets RemoteSolrException
 shard update error StdNode: 
 http://10.10.10.119:8980/solr/mycore/:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
  Illegal to have multiple roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:425)
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:401)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
 Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
 On the other server 10.10.10.119 we get following error
 org.apache.solr.common.SolrException: Illegal to have multiple roots (start 
 tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176)
   at 
 org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
   at 
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
   at 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
   at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at 
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
   at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
   at 
 org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
   at 
 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at 
 org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: com.ctc.wstx.exc.WstxParsingException: 

[jira] [Commented] (SOLR-5402) SolrCloud 4.5 bulk add errors in cloud setup

2013-10-30 Thread Michael Tracey (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809099#comment-13809099
 ] 

Michael Tracey commented on SOLR-5402:
--

I can confirm this issue on Jetty, both the bundled 8.x version from the 
example directory, and a 9.0.6 upgraded version of jetty with Solr 4.5.1.  I 
believe the issue is separate from the container.

 SolrCloud 4.5 bulk add errors in cloud setup
 

 Key: SOLR-5402
 URL: https://issues.apache.org/jira/browse/SOLR-5402
 Project: Solr
  Issue Type: Bug
  Components: SolrCloud
Affects Versions: 4.5, 4.5.1
Reporter: Sai Gadde
 Fix For: 4.6


 We use out of the box Solr 4.5.1 no customization done. If we merge documents 
 via SolrJ to a single server it is perfectly working fine.
 But as soon as we add another node to the cloud we are getting following 
 while merging documents. We merge about 500 at a time using SolrJ. These 500 
 documents in total are about few MB (1-3) in size.
 This is the error we are getting on the server (10.10.10.116 - IP is 
 irrelavent just for clarity)where merging is happening. 10.10.10.119 is the 
 new node here. This server gets RemoteSolrException
 shard update error StdNode: 
 http://10.10.10.119:8980/solr/mycore/:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
  Illegal to have multiple roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:425)
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:401)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
 Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
 On the other server 10.10.10.119 we get following error
 org.apache.solr.common.SolrException: Illegal to have multiple roots (start 
 tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176)
   at 
 org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
   at 
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
   at 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
   at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at 
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
   at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
   at 
 org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
   at 
 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at 
 org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal to have multiple 
 roots (start tag in epilog?).
  at [row,col {unknown-source}]: 

[jira] [Commented] (SOLR-5402) SolrCloud 4.5 bulk add errors in cloud setup

2013-10-30 Thread Mark Miller (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809341#comment-13809341
 ] 

Mark Miller commented on SOLR-5402:
---

Have you tried fewer than 500 docs at a time to see if the problem persists?

 SolrCloud 4.5 bulk add errors in cloud setup
 

 Key: SOLR-5402
 URL: https://issues.apache.org/jira/browse/SOLR-5402
 Project: Solr
  Issue Type: Bug
  Components: SolrCloud
Affects Versions: 4.5, 4.5.1
Reporter: Sai Gadde
 Fix For: 4.6


 We use out of the box Solr 4.5.1 no customization done. If we merge documents 
 via SolrJ to a single server it is perfectly working fine.
 But as soon as we add another node to the cloud we are getting following 
 while merging documents. We merge about 500 at a time using SolrJ. These 500 
 documents in total are about few MB (1-3) in size.
 This is the error we are getting on the server (10.10.10.116 - IP is 
 irrelavent just for clarity)where merging is happening. 10.10.10.119 is the 
 new node here. This server gets RemoteSolrException
 shard update error StdNode: 
 http://10.10.10.119:8980/solr/mycore/:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
  Illegal to have multiple roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:425)
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:401)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
 Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
 On the other server 10.10.10.119 we get following error
 org.apache.solr.common.SolrException: Illegal to have multiple roots (start 
 tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176)
   at 
 org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
   at 
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
   at 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
   at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at 
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
   at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
   at 
 org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
   at 
 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at 
 org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal to have multiple 
 roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12369]
   at 
 com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:630)
   at 

[jira] [Commented] (SOLR-5402) SolrCloud 4.5 bulk add errors in cloud setup

2013-10-30 Thread Michael Tracey (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809355#comment-13809355
 ] 

Michael Tracey commented on SOLR-5402:
--

Yes, I've found that a 50 docs will trigger the problem.

 SolrCloud 4.5 bulk add errors in cloud setup
 

 Key: SOLR-5402
 URL: https://issues.apache.org/jira/browse/SOLR-5402
 Project: Solr
  Issue Type: Bug
  Components: SolrCloud
Affects Versions: 4.5, 4.5.1
Reporter: Sai Gadde
 Fix For: 4.6


 We use out of the box Solr 4.5.1 no customization done. If we merge documents 
 via SolrJ to a single server it is perfectly working fine.
 But as soon as we add another node to the cloud we are getting following 
 while merging documents. We merge about 500 at a time using SolrJ. These 500 
 documents in total are about few MB (1-3) in size.
 This is the error we are getting on the server (10.10.10.116 - IP is 
 irrelavent just for clarity)where merging is happening. 10.10.10.119 is the 
 new node here. This server gets RemoteSolrException
 shard update error StdNode: 
 http://10.10.10.119:8980/solr/mycore/:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
  Illegal to have multiple roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:425)
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:401)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
 Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
 On the other server 10.10.10.119 we get following error
 org.apache.solr.common.SolrException: Illegal to have multiple roots (start 
 tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176)
   at 
 org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
   at 
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
   at 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
   at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at 
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
   at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
   at 
 org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
   at 
 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at 
 org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal to have multiple 
 roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12369]
   at 
 com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:630)
   at 

[jira] [Commented] (SOLR-5402) SolrCloud 4.5 bulk add errors in cloud setup

2013-10-30 Thread Sai Gadde (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809833#comment-13809833
 ] 

Sai Gadde commented on SOLR-5402:
-

I even tried adding 1 document at a time. Still the problem is present.

If there is only one server there no errors, everything works fine. updating 
even 500 documents is fine. 

But when another node comes online SolrCmdDistributor gets these exceptions 
from remote server response. New Solr node also prints it's own error stack as 
explained in the report above.

This exact same setup works without issues in 4.4.0. I tried with both 4.5.0 
and 4.5.1 and both produce same errors

 SolrCloud 4.5 bulk add errors in cloud setup
 

 Key: SOLR-5402
 URL: https://issues.apache.org/jira/browse/SOLR-5402
 Project: Solr
  Issue Type: Bug
  Components: SolrCloud
Affects Versions: 4.5, 4.5.1
Reporter: Sai Gadde
 Fix For: 4.6


 We use out of the box Solr 4.5.1 no customization done. If we merge documents 
 via SolrJ to a single server it is perfectly working fine.
 But as soon as we add another node to the cloud we are getting following 
 while merging documents. We merge about 500 at a time using SolrJ. These 500 
 documents in total are about few MB (1-3) in size.
 This is the error we are getting on the server (10.10.10.116 - IP is 
 irrelavent just for clarity)where merging is happening. 10.10.10.119 is the 
 new node here. This server gets RemoteSolrException
 shard update error StdNode: 
 http://10.10.10.119:8980/solr/mycore/:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
  Illegal to have multiple roots (start tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:425)
   at 
 org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:401)
   at 
 org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:1)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
 Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
 On the other server 10.10.10.119 we get following error
 org.apache.solr.common.SolrException: Illegal to have multiple roots (start 
 tag in epilog?).
  at [row,col {unknown-source}]: [1,12468]
   at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:176)
   at 
 org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
   at 
 org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
   at 
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
   at 
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
   at 
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at 
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at 
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at 
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at 
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at 
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at 
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
   at 
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at 
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
   at 
 org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
   at 
 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at 
 org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at