Hi Daniel, Unfortunately, there likely isn't anything you can do to speed this up. The process of applying edits will be bound by the available read throughput on the edits stream and the CPU needs for processing each transaction. This is inherently a single-threaded process in the current implementation.
You can check progress by accessing the NameNode web UI and looking at the Startup Progress tab. This won't tell you exactly when it will be done, but the stats on the page will give you a sense for how quickly it is making progress. For the future, you might want to consider deploying monitoring to check the time of the last checkpoint, and alert if it goes beyond some threshold. This might give you a chance to react and fix it before the next NameNode restart. There are a few improvements planned in this area for increasing performance and reducing the chance of running without a recent checkpoint. If you're interested, you can watch the following issues in jira. https://issues.apache.org/jira/browse/HDFS-4923 https://issues.apache.org/jira/browse/HDFS-6353 https://issues.apache.org/jira/browse/HDFS-7609 Chris Nauroth Hortonworks http://hortonworks.com/ On Sat, Jan 17, 2015 at 9:17 AM, Daniel Haviv <danielru...@gmail.com> wrote: > Hi, > After restarting the namenode we discovered that there was no checkpoint > for quite a while. > We are waiting for all the changes to be applied to the fsimage, but it > seems like it will take hours. > > Is there something we can do to expedite the process? Increases > parallelism? Something at all? > > Thanks, > Daniel > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
