RE: Datanode doesn't start but there is no exception in the log
There is no shutdown message until I shutdown the DataNode. I used hostname of the machine that will run the DataNode and I now used the IP but there is no difference. Again the DataNode seems to freeze and the output at the log is the one I mentioned before. Subject: Re: Datanode doesn't start but there is no exception in the log From: highpoint...@gmail.com Date: Thu, 12 May 2011 23:59:02 -0600 To: common-user@hadoop.apache.org Have you defined the IP of the DN in the slaves file? Sent from my iPhone On May 12, 2011, at 7:27 PM, Bharath Mundlapudi bharathw...@yahoo.com wrote: Is that all the messages in the datanode log? Do you see any SHUTDOWN message also? -Bharath From: Panayotis Antonopoulos antonopoulos...@hotmail.com To: common-user@hadoop.apache.org Sent: Thursday, May 12, 2011 6:07 PM Subject: Datanode doesn't start but there is no exception in the log Hello, I am trying to set up Hadoop HDFS in a cluster for the first time. So far I was using pseudo-distributed mode on my PC at home and everything was working perfectly. Tha NameNode starts but the DataNode doesn't start and the log contains the following: 2011-05-13 04:01:13,663 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = clone1/147.102.4.129 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.2-cdh3u0 STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'hudson' on Fri Mar 25 19:56:23 PDT 2011 / 2011-05-13 04:01:14,019 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing. 2011-05-13 04:01:14,143 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean 2011-05-13 04:01:14,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010 2011-05-13 04:01:14,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s 2011-05-13 04:01:14,206 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2011-05-13 04:01:14,272 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075 2011-05-13 04:01:14,278 INFO org.mortbay.log: jetty-6.1.26 2011-05-13 04:01:14,567 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075 2011-05-13 04:01:14,570 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null 2011-05-13 04:01:14,976 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020 2011-05-13 04:01:14,978 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020 2011-05-13 04:01:14,981 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020 2011-05-13 04:01:14,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(clone1:50010, storageID=, infoPort=50075, ipcPort=50020) Does anyone know what might be wrong?? Thank you in advance! Panagiotis
RE: Datanode doesn't start but there is no exception in the log
There is no other information in the log (although when I run it on my pc and it works, there is more information in the log) and also the web page of the namenode doesn't contain any live datanodes as it should. That's why I said it freezes... I have no idea what is going on... Please if anyone can help because it is really important to make it work as soon as possible (for my diploma thesis) and I really have no idea what might go wrong. Subject: Re: Datanode doesn't start but there is no exception in the log From: highpoint...@gmail.com Date: Fri, 13 May 2011 08:22:53 -0600 To: common-user@hadoop.apache.org When you say freeze you mean there is nothing rolling in the log? Sent from my iPhone On May 13, 2011, at 2:28 AM, Panayotis Antonopoulos antonopoulos...@hotmail.com wrote: There is no shutdown message until I shutdown the DataNode. I used hostname of the machine that will run the DataNode and I now used the IP but there is no difference. Again the DataNode seems to freeze and the output at the log is the one I mentioned before. Subject: Re: Datanode doesn't start but there is no exception in the log From: highpoint...@gmail.com Date: Thu, 12 May 2011 23:59:02 -0600 To: common-user@hadoop.apache.org Have you defined the IP of the DN in the slaves file? Sent from my iPhone On May 12, 2011, at 7:27 PM, Bharath Mundlapudi bharathw...@yahoo.com wrote: Is that all the messages in the datanode log? Do you see any SHUTDOWN message also? -Bharath From: Panayotis Antonopoulos antonopoulos...@hotmail.com To: common-user@hadoop.apache.org Sent: Thursday, May 12, 2011 6:07 PM Subject: Datanode doesn't start but there is no exception in the log Hello, I am trying to set up Hadoop HDFS in a cluster for the first time. So far I was using pseudo-distributed mode on my PC at home and everything was working perfectly. Tha NameNode starts but the DataNode doesn't start and the log contains the following: 2011-05-13 04:01:13,663 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = clone1/147.102.4.129 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.2-cdh3u0 STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'hudson' on Fri Mar 25 19:56:23 PDT 2011 / 2011-05-13 04:01:14,019 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing. 2011-05-13 04:01:14,143 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean 2011-05-13 04:01:14,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010 2011-05-13 04:01:14,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s 2011-05-13 04:01:14,206 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2011-05-13 04:01:14,272 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075 2011-05-13 04:01:14,278 INFO org.mortbay.log: jetty-6.1.26 2011-05-13 04:01:14,567 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075 2011-05-13 04:01:14,570 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null 2011-05-13 04:01:14,976 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020 2011-05-13 04:01:14,978 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020 2011-05-13 04:01:14,981 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020 2011-05-13 04:01:14,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(clone1:50010, storageID=, infoPort=50075, ipcPort=50020) Does anyone know what might be wrong?? Thank you in advance! Panagiotis
RE: Datanode doesn't start but there is no exception in the log
(TCPTransport.java:341) at java.lang.Thread.run(Thread.java:662) Low Memory Detector daemon prio=10 tid=0x7f50f8025000 nid=0x18e1 runnable [0x] java.lang.Thread.State: RUNNABLE C2 CompilerThread1 daemon prio=10 tid=0x7f50f8022800 nid=0x18e0 waiting on condition [0x] java.lang.Thread.State: RUNNABLE C2 CompilerThread0 daemon prio=10 tid=0x7f50f801f800 nid=0x18df waiting on condition [0x] java.lang.Thread.State: RUNNABLE Signal Dispatcher daemon prio=10 tid=0x7f50f801d800 nid=0x18de runnable [0x] java.lang.Thread.State: RUNNABLE Finalizer daemon prio=10 tid=0x7f50f8001000 nid=0x18dd in Object.wait() [0x7f50fc808000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xec49c0f0 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked 0xec49c0f0 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Reference Handler daemon prio=10 tid=0x40181800 nid=0x18dc in Object.wait() [0x7f50fc909000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xec4923d8 (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked 0xec4923d8 (a java.lang.ref.Reference$Lock) main prio=10 tid=0x40115000 nid=0x18d2 runnable [0x7f5101d2e000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:220) at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:493) at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:117) at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114) at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171) - locked 0xeb430cc0 (a sun.security.provider.SecureRandom) at java.security.SecureRandom.nextBytes(SecureRandom.java:433) - locked 0xeb430f60 (a java.security.SecureRandom) at java.security.SecureRandom.next(SecureRandom.java:455) at java.util.Random.nextInt(Random.java:257) at org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(DataNode.java:608) at org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:629) at org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1378) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1438) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1563) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1573) VM Thread prio=10 tid=0x4017b000 nid=0x18db runnable GC task thread#0 (ParallelGC) prio=10 tid=0x40128000 nid=0x18d3 runnable GC task thread#1 (ParallelGC) prio=10 tid=0x4012a000 nid=0x18d4 runnable GC task thread#2 (ParallelGC) prio=10 tid=0x4012c000 nid=0x18d5 runnable GC task thread#3 (ParallelGC) prio=10 tid=0x4012e000 nid=0x18d6 runnable GC task thread#4 (ParallelGC) prio=10 tid=0x4012f800 nid=0x18d7 runnable GC task thread#5 (ParallelGC) prio=10 tid=0x40131800 nid=0x18d8 runnable GC task thread#6 (ParallelGC) prio=10 tid=0x40133800 nid=0x18d9 runnable GC task thread#7 (ParallelGC) prio=10 tid=0x40135000 nid=0x18da runnable VM Periodic Task Thread prio=10 tid=0x7f50f8374000 nid=0x18f2 waiting on condition JNI global references: 1741 From: ha...@cloudera.com Date: Fri, 13 May 2011 21:58:41 +0530 Subject: Re: Datanode doesn't start but there is no exception in the log To: common-user@hadoop.apache.org Hello Panayotis, Could you please post a jstack output of your hung process to look into? $ jstack PID of DN # will do. 2011/5/13 Panayotis Antonopoulos antonopoulos...@hotmail.com: There is no other information in the log (although when I run it on my pc and it works, there is more information in the log) and also the web page of the namenode doesn't contain any live datanodes as it should. That's why I said it freezes... I have no idea what is going on... Please if anyone can help because it is really important to make it work as soon as possible (for my diploma thesis) and I really have no idea what might go wrong. Subject: Re: Datanode doesn't start but there is no exception in the log From: highpoint...@gmail.com Date: Fri, 13 May 2011 08:22:53 -0600 To: common-user@hadoop.apache.org When you say freeze you mean there is nothing rolling in the log
RE: Datanode doesn't start but there is no exception in the log
I have been waiting for hours to see if it will ever start but it doesn't. I will check the links you sent me. Thanks again for your help!!! From: ha...@cloudera.com Date: Fri, 13 May 2011 23:18:40 +0530 Subject: Re: Datanode doesn't start but there is no exception in the log To: common-user@hadoop.apache.org Actually, only the last mentioned stack matters. Also see: https://issues.apache.org/jira/browse/HDFS-1835 On Fri, May 13, 2011 at 11:15 PM, Harsh J ha...@cloudera.com wrote: Hey, 2011/5/13 Panayotis Antonopoulos antonopoulos...@hotmail.com: 899599744@qtp-1416044437-1 - Acceptor0 SelectChannelConnector@0.0.0.0:50075 prio=10 tid=0x7f50f8414800 nid=0x1926 runnable [0x7f50f6eb1000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked 0xec473870 (a sun.nio.ch.Util$2) - locked 0xec473860 (a java.util.Collections$UnmodifiableSet) - locked 0xec4733c8 (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) and, 1620640756@qtp-1416044437-0 prio=10 tid=0x7f50f83f0800 nid=0x1925 in Object.wait() [0x7f50f75eb000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0xec473a88 (a org.mortbay.thread.QueuedThreadPool$PoolThread) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) - locked 0xec473a88 (a org.mortbay.thread.QueuedThreadPool$PoolThread) and, main prio=10 tid=0x40115000 nid=0x18d2 runnable [0x7f5101d2e000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:220) at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:493) at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:117) at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114) at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171) - locked 0xeb430cc0 (a sun.security.provider.SecureRandom) at java.security.SecureRandom.nextBytes(SecureRandom.java:433) - locked 0xeb430f60 (a java.security.SecureRandom) at java.security.SecureRandom.next(SecureRandom.java:455) at java.util.Random.nextInt(Random.java:257) at org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(DataNode.java:608) at org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:629) at org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1378) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1438) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1563) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1573) lead me to believe that you're running into this: http://search-hadoop.com/m/7Giae6vLWR1/securerandomsubj=Re+Entropy+Pool+and+HDFS+FS+Commands+Hanging+System Just give it some time and it should start up soon (you may generate some other activity on the DN to help it get some fresh entropy). Sometimes it may take up to a minute at start up. -- Harsh J -- Harsh J
RE: Datanode doesn't start but there is no exception in the log
I am using the cluster of the Lab of my university, so I have limited access to its settings. It consists of 10 nodes with Intel Xeon CPUs and GNU/Linux 2.6.38 Is there a way to solve the problem without changing the settings of the cluster? I am trying to use the patch that Harsh J sent me but I haven't managed to install it on hadoop properly yet. I have never installed a patch before... Date: Fri, 13 May 2011 15:09:38 -0400 Subject: Re: Datanode doesn't start but there is no exception in the log From: s...@basam.org To: common-user@hadoop.apache.org Sounds like your entropy pool is exhausted blocking the process. What sort of hardware/os combo are you running this on? Sridhar
RE: Datanode doesn't start but there is no exception in the log
I installed the patch: https://issues.apache.org/jira/browse/HDFS-1835 that Harsh J told me and now everything works great!!! I hope that this change won't create other problems. Thanks to everyone and especially to Harsh J!! I would never find the problem without your help!! From: antonopoulos...@hotmail.com To: common-user@hadoop.apache.org Subject: RE: Datanode doesn't start but there is no exception in the log Date: Sat, 14 May 2011 03:49:21 +0300 I am using the cluster of the Lab of my university, so I have limited access to its settings. It consists of 10 nodes with Intel Xeon CPUs and GNU/Linux 2.6.38 Is there a way to solve the problem without changing the settings of the cluster? I am trying to use the patch that Harsh J sent me but I haven't managed to install it on hadoop properly yet. I have never installed a patch before... Date: Fri, 13 May 2011 15:09:38 -0400 Subject: Re: Datanode doesn't start but there is no exception in the log From: s...@basam.org To: common-user@hadoop.apache.org Sounds like your entropy pool is exhausted blocking the process. What sort of hardware/os combo are you running this on? Sridhar
Question about InputSampler
Hello, I am writing a MR job where the distribution of the Keys emitted by the Map phase is not known beforehand and so I can't create the partitions for the TotalOrderPartitioner. I would like to sample those keys to create the partitions and then run the job that will process the whole input. Is the InputSampler the tool I need? I tried to use it but I think it doesn't use the mapper class to process the samples and then create the partitions, but it just creates the partitions from the input. Am I wrong? Thank you in advance! Pan
Datanode doesn't start but there is no exception in the log
Hello, I am trying to set up Hadoop HDFS in a cluster for the first time. So far I was using pseudo-distributed mode on my PC at home and everything was working perfectly. Tha NameNode starts but the DataNode doesn't start and the log contains the following: 2011-05-13 04:01:13,663 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = clone1/147.102.4.129 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.2-cdh3u0 STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'hudson' on Fri Mar 25 19:56:23 PDT 2011 / 2011-05-13 04:01:14,019 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing. 2011-05-13 04:01:14,143 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean 2011-05-13 04:01:14,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010 2011-05-13 04:01:14,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s 2011-05-13 04:01:14,206 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2011-05-13 04:01:14,272 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075 2011-05-13 04:01:14,278 INFO org.mortbay.log: jetty-6.1.26 2011-05-13 04:01:14,567 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075 2011-05-13 04:01:14,570 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null 2011-05-13 04:01:14,976 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020 2011-05-13 04:01:14,978 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020 2011-05-13 04:01:14,981 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020 2011-05-13 04:01:14,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(clone1:50010, storageID=, infoPort=50075, ipcPort=50020) Does anyone know what might be wrong?? Thank you in advance! Panagiotis
How to make changes in Hadoop's classes visible
Hello, I am a beginner and I have downloaded Hadoop 0.20.2. I would like to make some changes in some of the classes of Hadoop and import them into hadoop-0.20.2-core.jar so that they are visible to the mapreduce programs. What is the best way to do this? What should I do if I find the changes at JIRA, for example if I want to change the LoadIncrementalHFiles class as mentioned at https://issues.apache.org/jira/browse/HBASE-3714? Is it the same procedure? Thank you in advance!! Panagiotis Antonopoulos.
ChainMapper and ChainReducer: Are the key/value pairs distributed to the nodes of the cluster before each Map phase?
Hello, Let' say we have a MR job that uses ChainMapper and ChainReducer like in the following diagram: Input-Map1-Map2-Reduce-Map3-Output The input is split and distributed to the nodes of the cluster before being processed by Map1 phase. Also, before the Reduce phase the key/value pairs are also distributed to the Reducers according to the Partitions made by the Partitioner. I expected that the same thing (distribution of the keys) would happen before Map2 and Map3 phases but after reading Pro Hadoop Book I strongly doubt it. I would like to ask you if the key/value pairs emitted by the Map1 phase (or those emitted by the Reduce phase) are distributed to the nodes of the cluster before being processed by the next Map phase, or if the output of the Map1 phase (or Reduce phase) is immediately inserted to Map2 phase (or Map3 Phase) within the same node, without any distribution. Thank you in advance! Panagiotis Antonopoulos
