Re: Security issue: hadoop fs shell bypass authentication?
On Mar 5, 2010, at 4:49 PM, Allen Wittenauer wrote: On 3/5/10 1:57 PM, jiang licht licht_ji...@yahoo.com wrote: So, this means that hadoop fs shell does not require any authentication and can be fired from anywhere? There is no authentication/security layer in any released version of Hadoop. True, although we are busily adding it. *Smile* It is going into trunk and Yahoo is back porting all of the security work on top of the Yahoo 0.20 branch. The primary coding is done, it is undergoing QA now. The plan is to get it on to the alpha clusters by April, and production clusters by August. Although we haven't pushed the security branch out yet to our github repository, we should soon. (http://github.com/yahoo/hadoop-common ) -- Owen
Shuffle In Memory OutOfMemoryError
Hi all, We are seeing the following error in our reducers of a particular job: Error: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1508) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1408) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1261) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1195) After enough reducers fail the entire job fails. This error occurs regardless of whether mapred.compress.map.output is true. We were able to avoid the issue by reducing mapred.job.shuffle.input.buffer.percent to 20%. Shouldn't the framework via ShuffleRamManager.canFitInMemory and.ShuffleRamManager.reserve correctly detect the the memory available for allocation? I would think that with poor configuration settings (and default settings in particular) the job may not be as efficient, but wouldn't die. Here is some more context in the logs, I have attached the full reducer log here: http://gist.github.com/323746 2010-03-06 07:54:49,621 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 4191933 bytes (435311 raw bytes) into RAM from attempt_201003060739_0002_m_61_0 2010-03-06 07:54:50,222 INFO org.apache.hadoop.mapred.ReduceTask: Task attempt_201003060739_0002_r_00_0: Failed fetch #1 from attempt_201003060739_0002_m_000202_0 2010-03-06 07:54:50,223 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201003060739_0002_r_00_0 adding host hd37.dfs.returnpath.net to penalty box, next contact in 4 seconds 2010-03-06 07:54:50,223 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201003060739_0002_r_00_0: Got 1 map-outputs from previous failures 2010-03-06 07:54:50,223 FATAL org.apache.hadoop.mapred.TaskRunner: attempt_201003060739_0002_r_00_0 : Map output copy failure : java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1508) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1408) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1261) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1195) We tried this both in 0.20.1 and 0.20.2. We had hoped MAPREDUCE-1182 would address the issue in 0.20.2, but it did not. Does anyone have any comments or suggestions? Is this a bug I should file a JIRA for? Jacob Rideout Return Path
Re: how to get cluster-ips
I am using ec2 and dont see the slaves in $HADOOP_HOME/conf/slaves file. On Sat, Mar 6, 2010 at 9:33 PM, Ted Yu yuzhih...@gmail.com wrote: check conf/slaves file on master: http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29#conf.2Fslaves_.28master_only.29 On Fri, Mar 5, 2010 at 7:13 PM, prasenjit mukherjee pmukher...@quattrowireless.com wrote: Is there any way ( like hadoop-commandline or files ) to know ip address of all the cluster nodes ( from master )
Re: Security issue: hadoop fs shell bypass authentication?
Good to know and look forward to seeing next release of hadoop with such new security features... Thanks, -- Michael --- On Sat, 3/6/10, Owen O'Malley omal...@apache.org wrote: From: Owen O'Malley omal...@apache.org Subject: Re: Security issue: hadoop fs shell bypass authentication? To: common-user@hadoop.apache.org Date: Saturday, March 6, 2010, 2:20 AM On Mar 5, 2010, at 4:49 PM, Allen Wittenauer wrote: On 3/5/10 1:57 PM, jiang licht licht_ji...@yahoo.com wrote: So, this means that hadoop fs shell does not require any authentication and can be fired from anywhere? There is no authentication/security layer in any released version of Hadoop. True, although we are busily adding it. *Smile* It is going into trunk and Yahoo is back porting all of the security work on top of the Yahoo 0.20 branch. The primary coding is done, it is undergoing QA now. The plan is to get it on to the alpha clusters by April, and production clusters by August. Although we haven't pushed the security branch out yet to our github repository, we should soon. (http://github.com/yahoo/hadoop-common) -- Owen
Re: Security issue: hadoop fs shell bypass authentication?
IMO, we should handle the security part at system level. In this case, you can configure iptable to restrict the connections to namenode. On 03/07/2010 05:56 AM, jiang licht wrote: Good to know and look forward to seeing next release of hadoop with such new security features... � Thanks, -- Michael --- On Sat, 3/6/10, Owen O'Malleyomal...@apache.org wrote: From: Owen O'Malleyomal...@apache.org Subject: Re: Security issue: hadoop fs shell bypass authentication? To: common-user@hadoop.apache.org Date: Saturday, March 6, 2010, 2:20 AM On Mar 5, 2010, at 4:49 PM, Allen Wittenauer wrote: On 3/5/10 1:57 PM, jiang lichtlicht_ji...@yahoo.com wrote: So, this means that hadoop fs shell does not require any authentication and can be fired from anywhere? There is no authentication/security layer in any released version of Hadoop. True, although we are busily adding it. *Smile* It is going into trunk and Yahoo is back porting all of the security work on top of the Yahoo 0.20 branch. The primary coding is done, it is undergoing QA now. The plan is to get it on to the alpha clusters by April, and production clusters by August. Although we haven't pushed the security branch out yet to our github repository, we should soon. (http://github.com/yahoo/hadoop-common) -- Owen
Re: how to get cluster-ips
There are several ways to get slave ip address. ( Not sure if you can use all of these on Ec2 ) 1. hadoop dfsadmin -report shows you list of nodes and there status 2. Name node slaves page displays information about live nodes. 3. You can execute commands on slaves nodes using bin/slaves.sh - bin/slaves.sh /sbin/ifconfig | grep inet addr - Ravi On 3/6/10 9:15 AM, prasenjit mukherjee prasen@gmail.com wrote: I am using ec2 and dont see the slaves in $HADOOP_HOME/conf/slaves file. On Sat, Mar 6, 2010 at 9:33 PM, Ted Yu yuzhih...@gmail.com wrote: check conf/slaves file on master: http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29#conf.2Fslaves_.28master_only.29 On Fri, Mar 5, 2010 at 7:13 PM, prasenjit mukherjee pmukher...@quattrowireless.com wrote: Is there any way ( like hadoop-commandline or files ) to know ip address of all the cluster nodes ( from master ) Ravi --
Re: Security issue: hadoop fs shell bypass authentication?
The upcoming security will work with kerberos. actions like running a map reduce job will involve getting a kerberos ticket and passing it along. I have dodged kerberos for a long time and not looking forward to much more complexity.but it will almost certainly be a switchable on off config option. On 3/6/10, Huy Phan dac...@gmail.com wrote: IMO, we should handle the security part at system level. In this case, you can configure iptable to restrict the connections to namenode. On 03/07/2010 05:56 AM, jiang licht wrote: Good to know and look forward to seeing next release of hadoop with such new security features... � Thanks, -- Michael --- On Sat, 3/6/10, Owen O'Malleyomal...@apache.org wrote: From: Owen O'Malleyomal...@apache.org Subject: Re: Security issue: hadoop fs shell bypass authentication? To: common-user@hadoop.apache.org Date: Saturday, March 6, 2010, 2:20 AM On Mar 5, 2010, at 4:49 PM, Allen Wittenauer wrote: On 3/5/10 1:57 PM, jiang lichtlicht_ji...@yahoo.com wrote: So, this means that hadoop fs shell does not require any authentication and can be fired from anywhere? There is no authentication/security layer in any released version of Hadoop. True, although we are busily adding it. *Smile* It is going into trunk and Yahoo is back porting all of the security work on top of the Yahoo 0.20 branch. The primary coding is done, it is undergoing QA now. The plan is to get it on to the alpha clusters by April, and production clusters by August. Although we haven't pushed the security branch out yet to our github repository, we should soon. (http://github.com/yahoo/hadoop-common) -- Owen
Re: how to get cluster-ips
Thanks Ravi, it helped. BTW, only the first trick worked : hadoop dfsadmin -report | grep Name: | cut -d: -f2 2nd one may not be applicable as I need to automate this ( hence need a commandline utility ) 3rd approach didnt work, as the commands are getting ecxecuted only on the local slave-node and not on all the slaves. -Prasen On Sun, Mar 7, 2010 at 7:05 AM, Ravi Phulari rphul...@yahoo-inc.com wrote: There are several ways to get slave ip address. ( Not sure if you can use all of these on Ec2 ) hadoop dfsadmin -report shows you list of nodes and there status Name node slaves page displays information about live nodes. You can execute commands on slaves nodes using bin/slaves.sh – bin/slaves.sh /sbin/ifconfig | grep “inet addr” - Ravi On 3/6/10 9:15 AM, prasenjit mukherjee prasen@gmail.com wrote: I am using ec2 and dont see the slaves in $HADOOP_HOME/conf/slaves file. On Sat, Mar 6, 2010 at 9:33 PM, Ted Yu yuzhih...@gmail.com wrote: check conf/slaves file on master: http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29#conf.2Fslaves_.28master_only.29 On Fri, Mar 5, 2010 at 7:13 PM, prasenjit mukherjee pmukher...@quattrowireless.com wrote: Is there any way ( like hadoop-commandline or files ) to know ip address of all the cluster nodes ( from master ) Ravi --
Re: Shuffle In Memory OutOfMemoryError
I think there is mismatch (in ReduceTask.java) between: this.numCopiers = conf.getInt(mapred.reduce.parallel.copies, 5); and: maxSingleShuffleLimit = (long)(maxSize * MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION); where MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION is 0.25f because copiers = new ArrayListMapOutputCopier(numCopiers); so the total memory allocated for in-mem shuffle is 1.25 * maxSize A JIRA should be filed to correlate the constant 5 above and MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION. Cheers On Sat, Mar 6, 2010 at 8:31 AM, Jacob R Rideout apa...@jacobrideout.netwrote: Hi all, We are seeing the following error in our reducers of a particular job: Error: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1508) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1408) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1261) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1195) After enough reducers fail the entire job fails. This error occurs regardless of whether mapred.compress.map.output is true. We were able to avoid the issue by reducing mapred.job.shuffle.input.buffer.percent to 20%. Shouldn't the framework via ShuffleRamManager.canFitInMemory and.ShuffleRamManager.reserve correctly detect the the memory available for allocation? I would think that with poor configuration settings (and default settings in particular) the job may not be as efficient, but wouldn't die. Here is some more context in the logs, I have attached the full reducer log here: http://gist.github.com/323746 2010-03-06 07:54:49,621 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 4191933 bytes (435311 raw bytes) into RAM from attempt_201003060739_0002_m_61_0 2010-03-06 07:54:50,222 INFO org.apache.hadoop.mapred.ReduceTask: Task attempt_201003060739_0002_r_00_0: Failed fetch #1 from attempt_201003060739_0002_m_000202_0 2010-03-06 07:54:50,223 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201003060739_0002_r_00_0 adding host hd37.dfs.returnpath.net to penalty box, next contact in 4 seconds 2010-03-06 07:54:50,223 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201003060739_0002_r_00_0: Got 1 map-outputs from previous failures 2010-03-06 07:54:50,223 FATAL org.apache.hadoop.mapred.TaskRunner: attempt_201003060739_0002_r_00_0 : Map output copy failure : java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1508) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1408) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1261) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1195) We tried this both in 0.20.1 and 0.20.2. We had hoped MAPREDUCE-1182 would address the issue in 0.20.2, but it did not. Does anyone have any comments or suggestions? Is this a bug I should file a JIRA for? Jacob Rideout Return Path
Re: Security issue: hadoop fs shell bypass authentication?
I can feel that pain, Kerberos needs you to pull more hair from your head :) I worked on it a while back and now only remember bit of it. But anyway, a secured hadoop cluster can be created on top of a carefully designed and deployed network and firewall system anyway, that's what ppl are now using, so, no worry actually ... Thank, -- Michael --- On Sat, 3/6/10, Edward Capriolo edlinuxg...@gmail.com wrote: From: Edward Capriolo edlinuxg...@gmail.com Subject: Re: Security issue: hadoop fs shell bypass authentication? To: common-user@hadoop.apache.org Date: Saturday, March 6, 2010, 8:46 PM The upcoming security will work with kerberos. actions like running a map reduce job will involve getting a kerberos ticket and passing it along. I have dodged kerberos for a long time and not looking forward to much more complexity.but it will almost certainly be a switchable on off config option. On 3/6/10, Huy Phan dac...@gmail.com wrote: IMO, we should handle the security part at system level. In this case, you can configure iptable to restrict the connections to namenode. On 03/07/2010 05:56 AM, jiang licht wrote: Good to know and look forward to seeing next release of hadoop with such new security features... � Thanks, -- Michael --- On Sat, 3/6/10, Owen O'Malleyomal...@apache.org wrote: From: Owen O'Malleyomal...@apache.org Subject: Re: Security issue: hadoop fs shell bypass authentication? To: common-user@hadoop.apache.org Date: Saturday, March 6, 2010, 2:20 AM On Mar 5, 2010, at 4:49 PM, Allen Wittenauer wrote: On 3/5/10 1:57 PM, jiang lichtlicht_ji...@yahoo.com wrote: So, this means that hadoop fs shell does not require any authentication and can be fired from anywhere? There is no authentication/security layer in any released version of Hadoop. True, although we are busily adding it. *Smile* It is going into trunk and Yahoo is back porting all of the security work on top of the Yahoo 0.20 branch. The primary coding is done, it is undergoing QA now. The plan is to get it on to the alpha clusters by April, and production clusters by August. Although we haven't pushed the security branch out yet to our github repository, we should soon. (http://github.com/yahoo/hadoop-common) -- Owen
