[
https://issues.apache.org/jira/browse/LUCENE-1591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless reopened LUCENE-1591:
----------------------------------------
Alas, horribly, I'm hitting this bug again, with the 2.10.0 Xerces JAR
currently checked in.
I downloaded the latest XML dump from Wikipedia (en),
enwiki-20110115-pages-articles.xml, and after ~2.8M docs I hit this:
{noformat}
[java] 592.4 sec --> main Wrote 2807000 line docs
[java] 592.51 sec --> main Wrote 2808000 line docs
[java] 592.59 sec --> main Wrote 2809000 line docs
[java] 592.78 sec --> main Wrote 2810000 line docs
[java] Exception in thread "Thread-0" java.lang.RuntimeException:
org.apache.xerces.impl.io.MalformedByteSequenceException: Invalid byte 2 of
4-byte UTF-8 sequence.
[java] at
org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSource$Parser.run(EnwikiContentSource.java:197)
[java] at java.lang.Thread.run(Thread.java:619)
[java] Caused by:
org.apache.xerces.impl.io.MalformedByteSequenceException: Invalid byte 2 of
4-byte UTF-8 sequence.
[java] at
org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown
Source)
[java] at
org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
[java] at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown
Source)
[java] at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown
Source)
[java] at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
[java] at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
[java] at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
Source)
[java] at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
Source)
[java] at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
[java] at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown
Source)
[java] at
org.apache.lucene.benchmark.byTask.feeds.EnwikiContentSource$Parser.run(EnwikiContentSource.java:174)
[java] ... 1 more
[java] Caused by:
org.apache.xerces.impl.io.MalformedByteSequenceException: Invalid byte 2 of
4-byte UTF-8 sequence.
[java] at org.apache.xerces.impl.io.UTF8Reader.invalidByte(Unknown
Source)
[java] at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source)
[java] at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
[java] at org.apache.xerces.impl.XMLEntityScanner.scanContent(Unknown
Source)
[java] at
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanContent(Unknown
Source)
[java] ... 8 more
[java] ####################
[java] ### D O N E !!! ###
[java] ####################
{noformat}
I went back to the old patched Xerces JAR, and it got past that point just
fine...
> Enable bzip compression in benchmark
> ------------------------------------
>
> Key: LUCENE-1591
> URL: https://issues.apache.org/jira/browse/LUCENE-1591
> Project: Lucene - Java
> Issue Type: Improvement
> Components: contrib/benchmark
> Reporter: Shai Erera
> Assignee: Mark Miller
> Fix For: 2.9
>
> Attachments: LUCENE-1591.patch, LUCENE-1591.patch, LUCENE-1591.patch,
> LUCENE-1591.patch, LUCENE-1591.patch, LUCENE-1591.patch, LUCENE-1591.patch,
> LUCENE-1591.patch, commons-compress-dev20090413.jar,
> commons-compress-dev20090413.jar
>
>
> bzip compression can aid the benchmark package by not requiring extracting
> bzip files (such as enwiki) in order to index them. The plan is to add a
> config parameter bzip.compression=true/false and in the relevant tasks either
> decompress the input file or compress the output file using the bzip streams.
> It will add a dependency on ant.jar which contains two classes similar to
> GZIPOutputStream and GZIPInputStream which compress/decompress files using
> the bzip algorithm.
> bzip is known to be superior in its compression performance to the gzip
> algorithm (~20% better compression), although it does the
> compression/decompression a bit slower.
> I wil post a patch which adds this parameter and implement it in
> LineDocMaker, EnwikiDocMaker and WriteLineDoc task. Maybe even add the
> capability to DocMaker or some of the super classes, so it can be inherited
> by all sub-classes.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
