Since my original implementation is still the fastest, and the newer implementations do not improve the speed of the application, then I must assume that the process is slow because of the XSLT transformations themselves. These transformations are straight-forward:
# transform the document and save it
my $doc = $parser->parse_file($file);
my $results = $stylesheet->transform($doc);
my $html_file = "$HTML_DIR/$id.html";
open OUT, "> $html_file";
print OUT $stylesheet->output_string($results);
close OUT;
# convert the HTML to plain text and save it
my $html = parse_htmlfile($html_file);
my $text_file = "$TEXT_DIR/$id.txt";
open OUT, "> $text_file";
print OUT $formatter->format($html);
close OUT;
Can you save some time by not re-parsing the HTML file? I haven't used the parse HTML feature of LibXML, but doesn't it produce the exact same kind of XML Document object? If so, you already have a copy in $results in the first part of the code, so you shouldn't need to go back and re-parse the file you just created, since $html should be identical, or at least functionally identical, to $results.
I don't know whether or not you are already doing this, but you might be able to save a lot of time if you don't re-parse documents and stylesheets, re-instantiate XML parsers, and so forth. Ideally, you would call XML::LibXML->new and XML::LibXSLT->new once at the beginning of the script, immediately followed by creating a $stylesheet that contains the parsed stylesheet that you can then apply to each document in the batch. You can then parse each source XML document once and perform all of your operations on it in one go. Your script could then look something vaguely like:
my $xml_parser = XML::LibXML->new; my $xslt_parser = XML::LibXSLT->new;
my $xslt_doc = $xml_parser->parse_file ('stylesheet.xsl');
my $stylesheet = $xslt_parser->parse_stylesheet ($xslt_doc);foreach my $file (@files_to_process) {
# Parse the document
my $original_doc = $xml_parser->parse_file ($file); # Transform to HTML
my $html_doc = $stylesheet->transform ($original_doc);
my $html_file = my_filenaming_algorithm ($file);
$html_doc->toFile ($html_file); # Transform the newly-transformed HTML (or XHTML) to plain text
open TEXT_OUT (">$text_file");
print TEXT_OUT $formatter->format ($html_doc);
close TEXT_OUT; # Grab selected information from the TEI header
my ($header) = $html_doc->findnodes('teiHeader');
my $author = $header->findvalue('fileDesc/titleStmt/author');
my $title = $header->findvalue('fileDesc/titleStmt/title');
my $id =$header->findvalue('fileDesc/publicationStmt/idno');
do_something_with_my_data ($author, $title, $id);
}That way, you only instantiate a parser once, you only parse the XML->HTML stylesheet once, and you only parse each XML document once. I don't know how much of this you are doing already, but eliminating unnecessary parsing could speed things up a fair bit.
I think the speed of the XSLT process depends a lot on how complex the stylesheet is. I have a script that parses XML documents and creates secondary XML documents which contain a small subset of the original data (with some fields amalgamated and otherwise massaged) and it takes maybe 20-40 minutes to batch about 4 or 5 gigabytes of data in about 8500 files. My original documents are quite large and numerous, but the derived documents are only about 1 KB or so, and the structure of the original is reasonably simple. The stylesheet itself is only about 100 lines, though the stylesheet rules do seem to include a lot of 'or' clauses in them. I don't know how complex your input files and transformations are compared to mine, or how fast your computer is, but 96 seconds to process 1.5 MB does seem a little slow compared to what I am getting.
I hope some of that helps.
-- William Wueppelmann Electronic Systems Specialist Canadian Institute for Historical Microreproductions (CIHM) http://www.canadiana.org
