Yeah I believe your /tmp was probably misbehaving somehow (running out of space or otherwise). You could log a JIRA for the "null" seen in the log though, it shouldn't have done that and should've shown the real mount point.
On Fri, Jul 19, 2013 at 8:47 PM, Azuryy Yu <azury...@gmail.com> wrote: > this is not a bug. > > it has been documented. > > On Jul 19, 2013 10:13 PM, "Krishna Kishore Bonagiri" > <write2kish...@gmail.com> wrote: >> >> Hi Harsh, >> >> I have made my dfs.namenode.name.dir point to a subdirectory of my home, >> and I don't see this issue again. So, is this a bug that we need to log into >> JIRA? >> >> Thanks, >> Kishore >> >> >> On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote: >>> >>> > 2013-07-12 11:04:26,002 WARN >>> > org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space >>> > available on volume 'null' is 0, which is below the configured reserved >>> > amount 104857600 >>> >>> This is interesting. Its calling your volume "null", which may be more >>> of a superficial bug. >>> >>> What is your dfs.namenode.name.dir set to? From >>> /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp >>> is being used off of the out-of-box defaults. Could you try to set it >>> to a specific directory thats not on /tmp? >>> >>> On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri >>> <write2kish...@gmail.com> wrote: >>> > I don't have it in my hdfs-site.xml, in which case probably the default >>> > value is taken.. >>> > >>> > >>> > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <azury...@gmail.com> wrote: >>> >> >>> >> please check dfs.datanode.du.reserved in the hdfs-site.xml >>> >> >>> >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <adityaexal...@gmail.com> >>> >> wrote: >>> >>> >>> >>> Hi Krishna, >>> >>> >>> >>> Can you please send screenshots of namenode web UI. >>> >>> >>> >>> Thanks Aditya. >>> >>> >>> >>> >>> >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri >>> >>> <write2kish...@gmail.com> wrote: >>> >>>> >>> >>>> I have had enough space on the disk that is used, like around 30 >>> >>>> Gigs >>> >>>> >>> >>>> Thanks, >>> >>>> Kishore >>> >>>> >>> >>>> >>> >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla >>> >>>> <venkatarami.ne...@cloudwick.com> wrote: >>> >>>>> >>> >>>>> Hi, >>> >>>>> pls see the available space for NN storage directory. >>> >>>>> >>> >>>>> Thanks & Regards >>> >>>>> >>> >>>>> Venkat >>> >>>>> >>> >>>>> >>> >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri >>> >>>>> <write2kish...@gmail.com> wrote: >>> >>>>>> >>> >>>>>> Hi, >>> >>>>>> >>> >>>>>> I am doing no activity on my single node cluster which is using >>> >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by >>> >>>>>> itself after >>> >>>>>> a while. I was looking at the name node log and see many of these >>> >>>>>> kinds of >>> >>>>>> entries.. Can anything be interpreted from these? >>> >>>>>> >>> >>>>>> 2013-07-12 09:06:11,256 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log >>> >>>>>> segment at >>> >>>>>> 561 >>> >>>>>> 2013-07-12 09:07:11,290 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log >>> >>>>>> from >>> >>>>>> 9.70.137.114 >>> >>>>>> 2013-07-12 09:07:11,290 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit >>> >>>>>> logs >>> >>>>>> 2013-07-12 09:07:11,290 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log >>> >>>>>> segment 561 >>> >>>>>> 2013-07-12 09:07:11,291 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of >>> >>>>>> transactions: 2 >>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched >>> >>>>>> in Syncs: >>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14 >>> >>>>>> 2013-07-12 09:07:11,292 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of >>> >>>>>> transactions: 2 >>> >>>>>> Total time for transactions(ms): 1 Number of transactions batched >>> >>>>>> in Syncs: >>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15 >>> >>>>>> 2013-07-12 09:07:11,293 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: >>> >>>>>> Finalizing edits >>> >>>>>> file >>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 >>> >>>>>> -> >>> >>>>>> >>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562 >>> >>>>>> 2013-07-12 09:07:11,294 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log >>> >>>>>> segment at >>> >>>>>> 563 >>> >>>>>> 2013-07-12 09:08:11,397 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log >>> >>>>>> from >>> >>>>>> 9.70.137.114 >>> >>>>>> 2013-07-12 09:08:11,398 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit >>> >>>>>> logs >>> >>>>>> 2013-07-12 09:08:11,398 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log >>> >>>>>> segment 563 >>> >>>>>> 2013-07-12 09:08:11,399 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of >>> >>>>>> transactions: 2 >>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched >>> >>>>>> in Syncs: >>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11 >>> >>>>>> 2013-07-12 09:08:11,400 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of >>> >>>>>> transactions: 2 >>> >>>>>> Total time for transactions(ms): 2 Number of transactions batched >>> >>>>>> in Syncs: >>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12 >>> >>>>>> 2013-07-12 09:08:11,402 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: >>> >>>>>> Finalizing edits >>> >>>>>> file >>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 >>> >>>>>> -> >>> >>>>>> >>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564 >>> >>>>>> 2013-07-12 09:08:11,402 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log >>> >>>>>> segment at >>> >>>>>> 565 >>> >>>>>> 2013-07-12 09:09:11,440 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log >>> >>>>>> from >>> >>>>>> 9.70.137.114 >>> >>>>>> 2013-07-12 09:09:11,440 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit >>> >>>>>> logs >>> >>>>>> 2013-07-12 09:09:11,440 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log >>> >>>>>> segment 565 >>> >>>>>> 2013-07-12 09:09:11,440 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of >>> >>>>>> transactions: 2 >>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched >>> >>>>>> in Syncs: >>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13 >>> >>>>>> 2013-07-12 09:09:11,441 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of >>> >>>>>> transactions: 2 >>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched >>> >>>>>> in Syncs: >>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13 >>> >>>>>> >>> >>>>>> >>> >>>>>> And after sometime it said: >>> >>>>>> >>> >>>>>> 2013-07-12 11:03:19,799 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log >>> >>>>>> segment at >>> >>>>>> 795 >>> >>>>>> 2013-07-12 11:04:19,826 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log >>> >>>>>> from >>> >>>>>> 9.70.137.114 >>> >>>>>> 2013-07-12 11:04:19,826 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit >>> >>>>>> logs >>> >>>>>> 2013-07-12 11:04:19,827 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log >>> >>>>>> segment 795 >>> >>>>>> 2013-07-12 11:04:19,827 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of >>> >>>>>> transactions: 2 >>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched >>> >>>>>> in Syncs: >>> >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12 >>> >>>>>> 2013-07-12 11:04:19,827 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of >>> >>>>>> transactions: 2 >>> >>>>>> Total time for transactions(ms): 0 Number of transactions batched >>> >>>>>> in Syncs: >>> >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12 >>> >>>>>> 2013-07-12 11:04:19,829 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: >>> >>>>>> Finalizing edits >>> >>>>>> file >>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 >>> >>>>>> -> >>> >>>>>> >>> >>>>>> /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796 >>> >>>>>> 2013-07-12 11:04:19,829 INFO >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log >>> >>>>>> segment at >>> >>>>>> 797 >>> >>>>>> 2013-07-12 11:04:26,002 WARN >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: >>> >>>>>> Space >>> >>>>>> available on volume 'null' is 0, which is below the configured >>> >>>>>> reserved >>> >>>>>> amount 104857600 >>> >>>>>> 2013-07-12 11:04:26,003 WARN >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low >>> >>>>>> on >>> >>>>>> available disk space. Entering safe mode. >>> >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: >>> >>>>>> STATE* Safe mode is ON. >>> >>>>>> Resources are low on NN. Please add or free up more resources then >>> >>>>>> turn off safe mode manually. NOTE: If you turn off safe mode >>> >>>>>> before adding >>> >>>>>> resources, the NN will immediately return to safe mode.. >>> >>>>>> 2013-07-12 11:04:31,004 WARN >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: >>> >>>>>> Space >>> >>>>>> available on volume 'null' is 0, which is below the configured >>> >>>>>> reserved >>> >>>>>> amount 104857600 >>> >>>>>> 2013-07-12 11:04:31,005 WARN >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low >>> >>>>>> on >>> >>>>>> available disk space. Already in safe mode. >>> >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: >>> >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free >>> >>>>>> up more >>> >>>>>> resources then turn off safe mode manually. NOTE: If you turn >>> >>>>>> off safe >>> >>>>>> mode before adding resources, the NN will immediately return to >>> >>>>>> safe mode.. >>> >>>>>> 2013-07-12 11:04:36,005 WARN >>> >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: >>> >>>>>> Space >>> >>>>>> available on volume 'null' is 0, which is below the configured >>> >>>>>> reserved >>> >>>>>> amount 104857600 >>> >>>>>> >>> >>>>>> >>> >>>>>> Thanks, >>> >>>>>> Kishore >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> N Venkata Rami Reddy >>> >>>>> Hadoop Admin >>> >>>>> Cloudwick Technologies >>> >>>> >>> >>>> >>> >>> >>> > >>> >>> >>> >>> -- >>> Harsh J >> >> > -- Harsh J
