org.apache.hadoop.streaming.StreamXmlRecordReader
-------------------------------------------------
Key: HADOOP-3465
URL: https://issues.apache.org/jira/browse/HADOOP-3465
Project: Hadoop Core
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.17.0
Environment: java version "1.6.0_06"
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java HotSpot(TM) Client VM (build 10.0-b22, mixed mode, sharing)
Linux hadoop-master 2.6.24.7-92.fc8 #1 SMP Wed May 7 16:50:09 EDT 2008 i686
i686 i386 GNU/Linux
hadoop-0.17.0
Reporter: David Campbell
Fix For: 0.17.0
I downloaded and installed the 0.17.0 version this morning.
I'm trying to use the StreamXmlRecordReader to parse a file that is formatted
like this:
<results>
<row>
<FIELD1>value</FIELD1>
..... many fields.
</row>
</results>
Each logical row has about 1,371 characters in it.
I have the following settings in my job.
conf.set("stream.recordreader.begin", "<row>");
conf.set("stream.recordreader.end", "</row>");
conf.set("stream.recordreader.maxrec", "500000");
When I run my tests, the TaskTracker shows me a severely truncated row like
this:
Processing record=<row>
<FIELD1><![CDATA[]]></FIELD2>
<FIELD2><![CDATA[TL]]></FIELD2>
<FIELD3><![CDATA[0003779]]></FIELD3>
<FIELD4><![CDATA[ABCD]]></FIELD4>
I've tried setting the maxrec limits but even the default should be (as I read
the code) more than big enough to handle ~1,371 characters from <row> to </row>.
And as you might expect, the XML parser in my Mapper task blows up because most
of the <row> </row> is missing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
