[
https://issues.apache.org/jira/browse/HBASE-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-5210:
----------------------------------
Attachment: HBASE-5210-crazy-new-getRandomFilename.patch
Patch for discussion.
> HFiles are missing from an incremental load
> -------------------------------------------
>
> Key: HBASE-5210
> URL: https://issues.apache.org/jira/browse/HBASE-5210
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.90.2
> Environment: HBase 0.90.2 with Hadoop-0.20.2 (with durable sync).
> RHEL 2.6.18-164.15.1.el5. 4 node cluster (1 master, 3 slaves)
> Reporter: Lawrence Simpson
> Attachments: HBASE-5210-crazy-new-getRandomFilename.patch
>
>
> We run an overnight map/reduce job that loads data from an external source
> and adds that data to an existing HBase table. The input files have been
> loaded into hdfs. The map/reduce job uses the HFileOutputFormat (and the
> TotalOrderPartitioner) to create HFiles which are subsequently added to the
> HBase table. On at least two separate occasions (that we know of), a range
> of output would be missing for a given day. The range of keys for the
> missing values corresponded to those of a particular region. This implied
> that a complete HFile somehow went missing from the job. Further
> investigation revealed the following:
> * Two different reducers (running in separate JVMs and thus separate class
> loaders)
> * in the same server can end up using the same file names for their
> * HFiles. The scenario is as follows:
> * 1. Both reducers start near the same time.
> * 2. The first reducer reaches the point where it wants to write its
> first file.
> * 3. It uses the StoreFile class which contains a static Random
> object
> * which is initialized by default using a timestamp.
> * 4. The file name is generated using the random number generator.
> * 5. The file name is checked against other existing files.
> * 6. The file is written into temporary files in a directory named
> * after the reducer attempt.
> * 7. The second reduce task reaches the same point, but its
> StoreClass
> * (which is now in the file system's cache) gets loaded within the
> * time resolution of the OS and thus initializes its Random()
> * object with the same seed as the first task.
> * 8. The second task also checks for an existing file with the name
> * generated by the random number generator and finds no conflict
> * because each task is writing files in its own temporary folder.
> * 9. The first task finishes and gets its temporary files committed
> * to the "real" folder specified for output of the HFiles.
> * 10. The second task then reaches its own conclusion and commits its
> * files (moveTaskOutputs). The released Hadoop code just
> overwrites
> * any files with the same name. No warning messages or anything.
> * The first task's HFiles just go missing.
> *
> * Note: The reducers here are NOT different attempts at the same
> * reduce task. They are different reduce tasks so data is
> * really lost.
> I am currently testing a fix in which I have added code to the Hadoop
> FileOutputCommitter.moveTaskOutputs method to check for a conflict with
> an existing file in the final output folder and to rename the HFile if
> needed. This may not be appropriate for all uses of FileOutputFormat.
> So I have put this into a new class which is then used by a subclass of
> HFileOutputFormat. Subclassing of FileOutputCommitter itself was a bit
> more of a problem due to private declarations.
> I don't know if my approach is the best fix for the problem. If someone
> more knowledgeable than myself deems that it is, I will be happy to share
> what I have done and by that time I may have some information on the
> results.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
