Hi, My configuration : hadoop-0.11.3 I ran m/r task. I monitoring http://namenode:50070/dfshealth.jsp page. Then one of datanode dead with followed hs_err log file. What's wrong?
====hs err log file ==== # # An unexpected error has been detected by HotSpot Virtual Machine: # # Internal Error (53484152454432554E54494D450E43505001A3), pid=12317, tid=0xa8dc00 # # Java VM: Java HotSpot(TM) 64-Bit Server VM (diablo-1.5.0_07-b01 mixed mode) --------------- T H R E A D --------------- Current thread (0x0000000000813000): JavaThread "[EMAIL PROTECTED]" daemon [_thread_in_Java, id=11066368] Stack: [0x00007ffffe5eb000,0x00007ffffe6eb000), sp=0x00007ffffe6e7c40, free space=1011k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x7ddc49] V [libjvm.so+0x476da6] V [libjvm.so+0x75a3ff] V [libjvm.so+0x6f4632] --------------- P R O C E S S --------------- Java Threads: ( => current thread ) =>0x0000000000813000 JavaThread "[EMAIL PROTECTED]" daemon [_thread_in_Java, id=11066368] 0x0000000000813400 JavaThread "[EMAIL PROTECTED]" daemon [_thread_in_native, id=11588608] 0x0000000000810800 JavaThread "[EMAIL PROTECTED]" daemon [_thread_in_native, id=11980800] 0x00000000007c2400 JavaThread "[EMAIL PROTECTED]" daemon [_thread_in_native, id=10022912] 0x000000000082e400 JavaThread "[EMAIL PROTECTED]" daemon [_thread_in_native, id=10009600] 0x0000000000621400 JavaThread "[EMAIL PROTECTED]" daemon [_thread_in_native, id=10189824] 0x0000000000621000 JavaThread "[EMAIL PROTECTED]" daemon [_thread_in_native, id=11063296] 0x0000000000b14400 JavaThread "[EMAIL PROTECTED]" daemon [_thread_in_native, id=11619328] 0x0000000000b0dc00 JavaThread "DataNode: [/home2/gfs/filesystem/data,/home/gfs/filesystem/data]" daemon [_thread_blocked, id=11616256] 0x0000000000afb000 JavaThread "SocketListener0-1" [_thread_blocked, id=11515904] 0x0000000000af9800 JavaThread "SocketListener0-0" [_thread_blocked, id=11508736] 0x0000000000af9000 JavaThread "Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=50075]" [_thread_in_native, id=11506688] 0x0000000000b33000 JavaThread "SessionScavenger" daemon [_thread_blocked, id=11744256] 0x000000000088c800 JavaThread "org.apache.hadoop.io.ObjectWritable Connection Culler" daemon [_thread_blocked, id=8965120] 0x00000000006f5400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=7319552] 0x00000000006ef400 JavaThread "CompilerThread1" daemon [_thread_blocked, id=7294976] 0x00000000006e9400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=7270400] 0x00000000006be400 JavaThread "AdapterThread" daemon [_thread_blocked, id=7245824] 0x00000000006a8400 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=7069696] 0x000000000069d800 JavaThread "Finalizer" daemon [_thread_blocked, id=6979584] 0x000000000069d000 JavaThread "Reference Handler" daemon [_thread_blocked, id=6935552] 0x0000000000527000 JavaThread "main" [_thread_blocked, id=5332992] Other Threads: 0x000000000065f600 VMThread [id=6470656] 0x000000000051dc00 WatcherThread [id=7344128] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap PSYoungGen total 2944K, used 2313K [0x0000000836860000, 0x0000000836b70000, 0x000000084b5b0000) eden space 2752K, 80% used [0x0000000836860000,0x0000000836a8a4e8,0x0000000836b10000) from space 192K, 50% used [0x0000000836b40000,0x0000000836b58000,0x0000000836b70000) to space 192K, 0% used [0x0000000836b10000,0x0000000836b10000,0x0000000836b40000) PSOldGen total 10240K, used 7923K [0x000000080cdb0000, 0x000000080d7b0000, 0x0000000836860000) object space 10240K, 77% used [0x000000080cdb0000,0x000000080d56cd60,0x000000080d7b0000) PSPermGen total 21504K, used 10830K [0x0000000807bb0000, 0x00000008090b0000, 0x000000080cdb0000) object space 21504K, 50% used [0x0000000807bb0000,0x0000000808643868,0x00000008090b0000) PSPermGen total 21504K, used 10830K [0x0000000807bb0000, 0x00000008090b0000, 0x000000080cdb0000) object space 21504K, 50% used [0x0000000807bb0000,0x0000000808643868,0x00000008090b0000) Dynamic libraries: 0x0000000000400000 /home/toolkit/java/bin/java 0x000000080063b000 /lib/libz.so.3 0x000000080074f000 /lib/libpthread.so.2 0x000000080087a000 /lib/libc.so.6 0x0000000800a89000 /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/server/libjvm.so 0x0000000801516000 /usr/lib/libstdc++.so.5 0x000000080170d000 /lib/libm.so.4 0x0000000801829000 /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/native_threads/libhpi.so 0x0000000801935000 /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libverify.so 0x0000000801a44000 /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libjava.so 0x0000000801b6c000 /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libzip.so 0x000000084c6f8000 /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libnet.so 0x000000084c80b000 /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libnio.so 0x000000084c912000 /home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/libmanagement.so 0x000000080050a000 /libexec/ld-elf.so.1 VM Arguments: jvm_args: -Xmx1000m -Dhadoop.log.dir=/home/nutch/logs -Dhadoop.log.file=hadoop-nutch-datanode-nutch1.log -Dhadoop.home.dir=/home/nutch -Dhadoop.id.str=nutch -Dhadoop.root.logger=INFO,DRFA -Djava.library.path=/home/nutch/lib/native/FreeBSD-amd64-64:/home/nutch/lib:/home/toolkit/iconv/lib:/usr/local/lib:/home/nutch/lib:/usr/local/lib:/usr/local/lib:/home/nutch/lib java_command: org.apache.hadoop.dfs.DataNode Launcher Type: SUN_STANDARD Environment Variables: JAVA_HOME=/home/toolkit/java PATH=/home/toolkit/java/bin:/home/toolkit/ant/bin:/bin:/usr/local/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/sbin: LD_LIBRARY_PATH=/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64/server:/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/lib/amd64:/home/toolkit/diablo-jdk1.5.0_07_amd64/jre/../lib/amd64:/home/nutch/lib:/usr/local/lib SHELL=/bin/csh HOSTTYPE=FreeBSD OSTYPE=FreeBSD MACHTYPE=unknown Signal Handlers: SIGSEGV: [libjvm.so+0x7de650], sa_mask[0]=0xffffffff, sa_flags=0x00000002 SIGBUS: [libjvm.so+0x7de650], sa_mask[0]=0xffffffff, sa_flags=0x00000002 SIGFPE: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042 SIGPIPE: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042 SIGILL: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042 SIGUSR1: [libjvm.so+0x6f30c0], sa_mask[0]=0x00000000, sa_flags=0x00000040 SIGUSR2: [libjvm.so+0x6f3030], sa_mask[0]=0xffffffff, sa_flags=0x00000042 SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000002 SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGQUIT: [libjvm.so+0x6f1b30], sa_mask[0]=0xffffffff, sa_flags=0x00000002 SIGTERM: [libjvm.so+0x6f1b30], sa_mask[0]=0xffffffff, sa_flags=0x00000002 --------------- S Y S T E M --------------- OS:FreeBSD uname:FreeBSD 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Mon Feb 12 20:35:02 KST 2007 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/NUTCH amd64 rlimit: STACK 524288k, CORE 0k, NOFILE 11095 CPU:total 2 amd64 3dnow Memory: 4k page, physical 1856136k vm_info: Java HotSpot(TM) 64-Bit Server VM (diablo-1.5.0_07-b01) for freebsd-amd64, built on Sep 24 2006 16:09:01 by root with gcc 3.4.4 [FreeBSD] 20050518
