2011-01-06 15:52:00,814 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.10 cmd=open src=/logs/remote.log.01.20110105.2045.lzo dst=null perm=null 2011-01-06 15:52:00,881 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.12 cmd=open src=/logs/remote.log.01.20110104.1815.lzo dst=null perm=null 2011-01-06 15:52:00,891 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.9 cmd=open src=/logs/remote.log.01.20110105.0015.lzo dst=null perm=null 2011-01-06 15:52:01,253 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.7 cmd=open src=/logs/remote.log.01.20110105.2035.lzo dst=null perm=null 2011-01-06 15:52:01,612 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.11 cmd=open src=/logs/remote.log.01.20110105.2145.lzo dst=null perm=null 2011-01-06 15:52:01,715 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.8 cmd=open src=/logs/remote.log.01.20110104.1915.lzo dst=null perm=null 2011-01-06 15:52:02,030 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.13 cmd=open src=/logs/remote.log.01.20110105.1235.lzo dst=null perm=null 2011-01-06 15:52:02,701 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask 10.11.138.52:50010 to delete blk_5975970115065306335_564427 blk_-4283643305091895835_564439 2011-01-06 15:52:02,701 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask 10.90.37.8:50010 to delete blk_7719330829554745618_564439 2011-01-06 15:52:02,701 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask 10.11.138.73:50010 to delete blk_5975970115065306335_564427 2011-01-06 15:52:02,701 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask 10.11.138.156:50010 to delete blk_-4283643305091895835_564439 blk_7719330829554745618_564439 2011-01-06 15:52:04,256 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.90.37.13 2011-01-06 15:52:04,494 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.12 cmd=open src=/logs/remote.log.01.20110105.1120.lzo dst=null perm=null 2011-01-06 15:52:04,529 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.7 cmd=open src=/logs/remote.log.01.20110104.1950.lzo dst=null perm=null 2011-01-06 15:52:04,563 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.9 cmd=open src=/logs/remote.log.01.20110105.0215.lzo dst=null perm=null 2011-01-06 15:52:05,255 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.11 cmd=open src=/logs/remote.log.01.20110105.2245.lzo dst=null perm=null 2011-01-06 15:52:05,261 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.10 cmd=open src=/logs/remote.log.01.20110105.0835.lzo dst=null perm=null 2011-01-06 15:52:06,960 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.8 cmd=open src=/logs/remote.log.01.20110104.0855.lzo dst=null perm=null 2011-01-06 15:52:07,147 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.io.IOException: Server returned HTTP response code: 503 for URL: http://10.90.37.13:50090/getimage?getimage=1 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1313) at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151) at org.apache.hadoop.hdfs.server.namenode.GetImageServlet.doGet(GetImageServlet.java:58) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2011-01-06 15:52:07,522 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=admin,users,admin ip=/10.90.37.10 cmd=open src=/logs/remote.log.01.20110104.2125.lzo dst=null perm=null The errors from the 2nn repeat frequently, I grabbed a snippet of the nn log from the most recent occurange (i.e. it's not the same occurance as my original post) From: suresh srinivas [mailto:srini30...@gmail.com] Sent: January 6, 2011 15:27 To: hdfs-user@hadoop.apache.org Subject: Re: Secondary Namenode doCheckpoint, FileNotFoundException Can you add namenode log around this time? --------------------------------------------------------------------- This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
