simplePostTool is just that, simple. It's intended to get you started. It is not a full-featured web crawler. As such, if you're encountering wonky web pages that are not well formed HTML there's no guarantee that it'll handle them gracefully.
Crawling websites is a pain, so if you require something robust I'd investigate Nutch (which integrates with Solr/Lucene) or similar. Best, Erick On Fri, Jul 15, 2016 at 9:01 AM, Musshorn, Kris T CTR USARMY RDECOM ARL (US) <kris.t.musshorn....@mail.mil> wrote: > CLASSIFICATION: UNCLASSIFIED > > How do I correct this error when running the simple post tool against a > website? > The tool successfully indexed for about 30 mins before throwing this error > and terminating. > > [Fatal Error] :642:15: XML document structures must start and end within the > same entity. > Exception in thread "main" java.lang.RuntimeException: > org.xml.sax.SAXParseException; lineNumber: 642; columnNumber: 15; XML > document structures must start and end within the same entity. > at > org.apache.solr.util.SimplePostTool$PageFetcher.getLinksFromWebPage(SimplePostTool.java:1219) > at > org.apache.solr.util.SimplePostTool.webCrawl(SimplePostTool.java:601) > at > org.apache.solr.util.SimplePostTool.webCrawl(SimplePostTool.java:618) > at > org.apache.solr.util.SimplePostTool.webCrawl(SimplePostTool.java:618) > at > org.apache.solr.util.SimplePostTool.webCrawl(SimplePostTool.java:618) > at > org.apache.solr.util.SimplePostTool.webCrawl(SimplePostTool.java:618) > at > org.apache.solr.util.SimplePostTool.postWebPages(SimplePostTool.java:548) > at > org.apache.solr.util.SimplePostTool.doWebMode(SimplePostTool.java:351) > at > org.apache.solr.util.SimplePostTool.execute(SimplePostTool.java:182) > at org.apache.solr.util.SimplePostTool.main(SimplePostTool.java:167) > Caused by: org.xml.sax.SAXParseException; lineNumber: 642; columnNumber: 15; > XML document structures must start and end within the same entity. > at > com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257) > at > com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339) > at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) > at > org.apache.solr.util.SimplePostTool.makeDom(SimplePostTool.java:1028) > at > org.apache.solr.util.SimplePostTool$PageFetcher.getLinksFromWebPage(SimplePostTool.java:1201) > ... 9 more > > Thanks, > Kris > > ~~~~~~~~~~~~~~~~~~~~~~~~~~ > Kris T. Musshorn > FileMaker Developer - Contractor - Catapult Technology Inc. > US Army Research Lab > Aberdeen Proving Ground > Application Management & Development Branch > 410-278-7251 > kris.t.musshorn....@mail.mil > ~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > > CLASSIFICATION: UNCLASSIFIED
