[ https://issues.apache.org/jira/browse/PIG-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999053#comment-12999053 ]
Alan Gates commented on PIG-1842: --------------------------------- >From reviewing the code it is not clear to me how this splits the XML file. >Let's say we have an XML file that looks like: {code} <a> <b> <c> </c> <c1> </c1> </b> </a> <a1> <b1> </b1> <b2> </b2> </a2> {code} and the split falls on line "</c1>". How far will split 1 read? It seems like it has to read to "</a>" or else the map processing split one will not be able to process this as a coherent document. Yet from the setting of maxBytesReadable on line 132 it looks to me like it won't read past the end point. How does split 2 know where to start? I don't see any code that is telling split 2 to fast forward to the point where split 1 ends. All the tests pass just fine. > Improve Scalability of the XMLLoader for large datasets such as wikipedia > ------------------------------------------------------------------------- > > Key: PIG-1842 > URL: https://issues.apache.org/jira/browse/PIG-1842 > Project: Pig > Issue Type: Improvement > Components: impl > Affects Versions: 0.7.0, 0.8.0, 0.9.0 > Reporter: Viraj Bhat > Assignee: Vivek Padmanabhan > Fix For: 0.7.0, 0.8.0, 0.9.0 > > Attachments: PIG-1842_1.patch, PIG-1842_2.patch > > > The current XMLLoader for Pig, does not work well for large datasets such as > the wikipedia dataset. Each mapper reads in the entire XML file resulting in > extermely slow run times. > Viraj -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira