Error importing hbase table on new system
Hi, Iam trying to import a old backup to a new smaller system (just single node, to get the data out) when I use sudo -u hbase hbase -Dhbase.import.version=0.94 org.apache.hadoop.hbase.mapreduce.Import crawler /crawler_hbase/crawler I get this error in the tasks . Is this a permission problem? 2015-09-26 23:56:32,995 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:mapred (auth:SIMPLE) cause:java.io.IOException: keyvalues=NONE read 4096 bytes, should read 14279 2015-09-26 23:56:32,996 WARN org.apache.hadoop.mapred.Child: Error running child java.io.IOException: keyvalues=NONE read 4096 bytes, should read 14279 at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2221) at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483) at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapred.Child.main(Child.java:262) 2015-09-26 23:56:33,002 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task -- Håvard Wahl Kongsgård Data Scientist
Re: windows installation error
Hi, I am writing this email as a note for other users to resolve if they come across such issue. Found the solution by trying numerous building hadoop distributions. The problem is not due to hadoop distributions or versions obviously, It is because of building winutils id hadoop-common project. While building winutils, it was requiring cvtres.exe to be called to build the Windows Solution dependency(winutils.sln). So if you installed .net 4.5 framework, cvtres.exe was using a wrong version of dll dependency. What i did was to uninstall .net framework 4.5 and installing 4.0 framework which is using the right version dll file and helping maven package ... command to get over the issue and build hadoop successfully, Possibble other solutions which is reasoned by cvtres.exe are explained in stackoverflow and in this blog post and windows update reference site. http://stackoverflow.com/questions/10888391/error-link-fatal-error-lnk1123-failure-during-conversion-to-coff-file-inval https://aimslife.wordpress.com/2014/01/22/incremental-linking-lnk1123-error-in-vs2010-while-building-project/ http://blogs.msdn.com/b/heaths/archive/2011/04/01/visual-c-2010-sp1-compiler-update-for-the-windows-sdk-7-1.aspx Thanks in advance... On Sat, Sep 26, 2015 at 11:59 PM, Onder SEZGIN wrote: > Yes, the command i am running is > > >mvn -e package -Pdist,native-win -DskipTests -Dtar > > and i follow BUILDING.txt > > Önder > > On Sat, Sep 26, 2015 at 11:31 PM, Ted Yu wrote: > >> Have you specified the following profile ? >> >> -Pnative-win >> >> Please see 'Building on Windows' section in BUILDING.txt >> >> Details can be found in this commit: >> >> 638801cce16fc1dc3259c541dc30a599faaddda1 >> >> Cheers >> >> On Sat, Sep 26, 2015 at 6:51 AM, Onder SEZGIN >> wrote: >> >>> It was the latest release. >>> -X gives the same output because i had already supplied -e option while >>> running mvn package command. >>> >>> On Saturday, September 26, 2015, Ted Yu wrote: >>> Which release of hadoop are you installing ? Have you tried supplying -X switch to see debug logs ? Cheers On Sat, Sep 26, 2015 at 5:46 AM, Onder SEZGIN wrote: > Hi, > > I am trying to build hadoop on windows. > and i am getting the error below. > > Is there anyone who could get over the issue? > > Cheers > > [ERROR] Failed to execute goal > org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (c > ompile-ms-winutils) on project hadoop-common: Command execution > failed. Process > exited with an error: 1 (Exit value: 1) -> [Help 1] > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to > execute goal o > rg.codehaus.mojo:exec-maven-plugin:1.3.1:exec (compile-ms-winutils) on > project h > adoop-common: Command execution failed. > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor > .java:216) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor > .java:153) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor > .java:145) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje > ct(LifecycleModuleBuilder.java:84) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje > ct(LifecycleModuleBuilder.java:59) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBu > ild(LifecycleStarter.java:183) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(Lifecycl > eStarter.java:161) > at > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:317) > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:152) > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:555) > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214) > at org.apache.maven.cli.MavenCli.main(MavenCli.java:158) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. > java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces > sorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Laun > cher.java:289) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav > a:229) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La > uncher.java:415) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java: > 356) > Caused by: org.apache.maven.plugin.MojoExecutionException: Command > execution fai > led.
Re: Build error on windows
Hi, I am writing this email as a note for other users to resolve if they come across such issue. Found the solution by trying numerous building hadoop distributions. The problem is not due to hadoop distributions or versions obviously, It is because of building winutils id hadoop-common project. While building winutils, it was requiring cvtres.exe to be called to build the Windows Solution dependency(winutils.sln). So if you installed .net 4.5 framework, cvtres.exe was using a wrong version of dll dependency. What i did was to uninstall .net framework 4.5 and installing 4.0 framework which is using the right version dll file and helping maven package ... command to get over the issue and build hadoop successfully, Possibble other solutions which is reasoned by cvtres.exe are explained in stackoverflow and in this blog post and windows update reference site. http://stackoverflow.com/questions/10888391/error-link-fatal-error-lnk1123-failure-during-conversion-to-coff-file-inval https://aimslife.wordpress.com/2014/01/22/incremental-linking-lnk1123-error-in-vs2010-while-building-project/ http://blogs.msdn.com/b/heaths/archive/2011/04/01/visual-c-2010-sp1-compiler-update-for-the-windows-sdk-7-1.aspx Thanks in advance... On Sat, Sep 26, 2015 at 5:49 PM, Onder SEZGIN wrote: > Hi, > > I am trying to build hadoop on windows. > and i am getting the error below. > > Is there anyone who could get over the issue? > > Cheers > > [ERROR] Failed to execute goal > org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (c > ompile-ms-winutils) on project hadoop-common: Command execution failed. > Process > exited with an error: 1 (Exit value: 1) -> [Help 1] > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal o > rg.codehaus.mojo:exec-maven-plugin:1.3.1:exec (compile-ms-winutils) on > project h > adoop-common: Command execution failed. > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor > .java:216) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor > .java:153) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor > .java:145) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje > ct(LifecycleModuleBuilder.java:84) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje > ct(LifecycleModuleBuilder.java:59) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBu > ild(LifecycleStarter.java:183) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(Lifecycl > eStarter.java:161) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:317) > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:152) > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:555) > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214) > at org.apache.maven.cli.MavenCli.main(MavenCli.java:158) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. > java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces > sorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Laun > cher.java:289) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav > a:229) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La > uncher.java:415) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java: > 356) > Caused by: org.apache.maven.plugin.MojoExecutionException: Command > execution fai > led. > at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:303) > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(Default > BuildPluginManager.java:106) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor > .java:208) > ... 19 more > Caused by: org.apache.commons.exec.ExecuteException: Process exited with > an erro > r: 1 (Exit value: 1) > at > org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecut > or.java:402) > at > org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java: > 164) > at > org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:750) > > at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:292) > ... 21 more > [ERROR] > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please rea > d the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE > xception > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the
Re: Error importing hbase table on new system
Is the single node system secure ? Have you checked hdfs healthiness ? To which release of hbase were you importing ? Thanks > On Sep 27, 2015, at 3:06 AM, Håvard Wahl Kongsgård > wrote: > > Hi, Iam trying to import a old backup to a new smaller system (just > single node, to get the data out) > > when I use > > sudo -u hbase hbase -Dhbase.import.version=0.94 > org.apache.hadoop.hbase.mapreduce.Import crawler > /crawler_hbase/crawler > > I get this error in the tasks . Is this a permission problem? > > > 2015-09-26 23:56:32,995 ERROR > org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:mapred (auth:SIMPLE) > cause:java.io.IOException: keyvalues=NONE read 4096 bytes, should read > 14279 > 2015-09-26 23:56:32,996 WARN org.apache.hadoop.mapred.Child: Error running > child > java.io.IOException: keyvalues=NONE read 4096 bytes, should read 14279 > at > org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2221) > at > org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483) > at > org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76) > at > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) > at org.apache.hadoop.mapred.Child$4.run(Child.java:268) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > at org.apache.hadoop.mapred.Child.main(Child.java:262) > 2015-09-26 23:56:33,002 INFO org.apache.hadoop.mapred.Task: Runnning > cleanup for the task > > > > -- > Håvard Wahl Kongsgård > Data Scientist
Re: Error importing hbase table on new system
>Is the single node system secure ? No have not activated, just defaults the mapred conf. mapred.job.tracker rack3:8021 mapred.jobtracker.plugins org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin Comma-separated list of jobtracker plug-ins to be activated. jobtracker.thrift.address 0.0.0.0:9290 >>Have you checked hdfs healthiness ? sudo -u hdfs hdfs dfsadmin -report Configured Capacity: 2876708585472 (2.62 TB) Present Capacity: 1991514849280 (1.81 TB) DFS Remaining: 1648230617088 (1.50 TB) DFS Used: 343284232192 (319.71 GB) DFS Used%: 17.24% Under replicated blocks: 52 Blocks with corrupt replicas: 0 Missing blocks: 0 - Datanodes available: 1 (1 total, 0 dead) Live datanodes: Name: 127.0.0.1:50010 (localhost) Hostname: rack3 Decommission Status : Normal Configured Capacity: 2876708585472 (2.62 TB) DFS Used: 343284232192 (319.71 GB) Non DFS Used: 885193736192 (824.40 GB) DFS Remaining: 1648230617088 (1.50 TB) DFS Used%: 11.93% DFS Remaining%: 57.30% Last contact: Sun Sep 27 13:44:45 CEST 2015 >>To which release of hbase were you importing ? Hbase 0.94 (CHD 4) the new one is CHD 5.4 On Sun, Sep 27, 2015 at 1:32 PM, Ted Yu wrote: > Is the single node system secure ? > Have you checked hdfs healthiness ? > To which release of hbase were you importing ? > > Thanks > >> On Sep 27, 2015, at 3:06 AM, Håvard Wahl Kongsgård >> wrote: >> >> Hi, Iam trying to import a old backup to a new smaller system (just >> single node, to get the data out) >> >> when I use >> >> sudo -u hbase hbase -Dhbase.import.version=0.94 >> org.apache.hadoop.hbase.mapreduce.Import crawler >> /crawler_hbase/crawler >> >> I get this error in the tasks . Is this a permission problem? >> >> >> 2015-09-26 23:56:32,995 ERROR >> org.apache.hadoop.security.UserGroupInformation: >> PriviledgedActionException as:mapred (auth:SIMPLE) >> cause:java.io.IOException: keyvalues=NONE read 4096 bytes, should read >> 14279 >> 2015-09-26 23:56:32,996 WARN org.apache.hadoop.mapred.Child: Error running >> child >> java.io.IOException: keyvalues=NONE read 4096 bytes, should read 14279 >> at >> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2221) >> at >> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74) >> at >> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483) >> at >> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76) >> at >> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85) >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139) >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) >> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) >> at org.apache.hadoop.mapred.Child.main(Child.java:262) >> 2015-09-26 23:56:33,002 INFO org.apache.hadoop.mapred.Task: Runnning >> cleanup for the task >> >> >> >> -- >> Håvard Wahl Kongsgård >> Data Scientist -- Håvard Wahl Kongsgård Data Scientist
Re: Error importing hbase table on new system
Have you verified that the files to be imported are in HFilev2 format ? http://hbase.apache.org/book.html#_hfile_tool Cheers On Sun, Sep 27, 2015 at 4:47 AM, Håvard Wahl Kongsgård < haavard.kongsga...@gmail.com> wrote: > >Is the single node system secure ? > > No have not activated, just defaults > > the mapred conf. > > > > > > > > > > > mapred.job.tracker > > rack3:8021 > > > > > > > > > mapred.jobtracker.plugins > > org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin > > Comma-separated list of jobtracker plug-ins to be > activated. > > > > > > > > jobtracker.thrift.address > > 0.0.0.0:9290 > > > > > > > >>Have you checked hdfs healthiness ? > > > sudo -u hdfs hdfs dfsadmin -report > > Configured Capacity: 2876708585472 (2.62 TB) > > Present Capacity: 1991514849280 (1.81 TB) > > DFS Remaining: 1648230617088 (1.50 TB) > > DFS Used: 343284232192 (319.71 GB) > > DFS Used%: 17.24% > > Under replicated blocks: 52 > > Blocks with corrupt replicas: 0 > > Missing blocks: 0 > > > - > > Datanodes available: 1 (1 total, 0 dead) > > > Live datanodes: > > Name: 127.0.0.1:50010 (localhost) > > Hostname: rack3 > > Decommission Status : Normal > > Configured Capacity: 2876708585472 (2.62 TB) > > DFS Used: 343284232192 (319.71 GB) > > Non DFS Used: 885193736192 (824.40 GB) > > DFS Remaining: 1648230617088 (1.50 TB) > > DFS Used%: 11.93% > > DFS Remaining%: 57.30% > > Last contact: Sun Sep 27 13:44:45 CEST 2015 > > > >>To which release of hbase were you importing ? > > Hbase 0.94 (CHD 4) > > the new one is CHD 5.4 > > On Sun, Sep 27, 2015 at 1:32 PM, Ted Yu wrote: > > Is the single node system secure ? > > Have you checked hdfs healthiness ? > > To which release of hbase were you importing ? > > > > Thanks > > > >> On Sep 27, 2015, at 3:06 AM, Håvard Wahl Kongsgård < > haavard.kongsga...@gmail.com> wrote: > >> > >> Hi, Iam trying to import a old backup to a new smaller system (just > >> single node, to get the data out) > >> > >> when I use > >> > >> sudo -u hbase hbase -Dhbase.import.version=0.94 > >> org.apache.hadoop.hbase.mapreduce.Import crawler > >> /crawler_hbase/crawler > >> > >> I get this error in the tasks . Is this a permission problem? > >> > >> > >> 2015-09-26 23:56:32,995 ERROR > >> org.apache.hadoop.security.UserGroupInformation: > >> PriviledgedActionException as:mapred (auth:SIMPLE) > >> cause:java.io.IOException: keyvalues=NONE read 4096 bytes, should read > >> 14279 > >> 2015-09-26 23:56:32,996 WARN org.apache.hadoop.mapred.Child: Error > running child > >> java.io.IOException: keyvalues=NONE read 4096 bytes, should read 14279 > >> at > org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2221) > >> at > org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74) > >> at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483) > >> at > org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76) > >> at > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85) > >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139) > >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) > >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) > >> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) > >> at java.security.AccessController.doPrivileged(Native Method) > >> at javax.security.auth.Subject.doAs(Subject.java:415) > >> at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > >> at org.apache.hadoop.mapred.Child.main(Child.java:262) > >> 2015-09-26 23:56:33,002 INFO org.apache.hadoop.mapred.Task: Runnning > >> cleanup for the task > >> > >> > >> > >> -- > >> Håvard Wahl Kongsgård > >> Data Scientist > > > > -- > Håvard Wahl Kongsgård > Data Scientist >
RE: Problem running example (wrong IP address)
Thanks for sharing the logs. Problem is interesting..can you please post namenode logs and dual IP configurations(thinking problem with gateway while sending requests from 52.1 segment to 51.1 segment..) Thanks And RegardsBrahma Reddy Battula Date: Fri, 25 Sep 2015 12:19:00 -0500 Subject: Re: Problem running example (wrong IP address) From: dwmaill...@gmail.com To: user@hadoop.apache.org hadoop-master http://pastebin.com/yVF8vCYShadoop-data1 http://pastebin.com/xMEdf01ehadoop-data2 http://pastebin.com/prqd02eZ On Fri, Sep 25, 2015 at 11:53 AM, Brahma Reddy Battula wrote: sorry,I am not able to access the logs, could please post in paste bin or attach the 192.168.51.6( as your query is why different IP) DN logs and namenode logs here..? Thanks And RegardsBrahma Reddy Battula Date: Fri, 25 Sep 2015 11:16:55 -0500 Subject: Re: Problem running example (wrong IP address) From: dwmaill...@gmail.com To: user@hadoop.apache.org Brahma, Thanks for the reply. I'll keep this conversation here in the user list. The /etc/hosts file is identical on all three nodes hadoop@hadoop-data1:~$ cat /etc/hosts127.0.0.1 localhost192.168.51.4 hadoop-master 192.168.52.4 hadoop-data1192.168.52.6 hadoop-data2 hadoop@hadoop-data2:~$ cat /etc/hosts127.0.0.1 localhost192.168.51.4 hadoop-master 192.168.52.4 hadoop-data1192.168.52.6 hadoop-data2 hadoop@hadoop-master:~$ cat /etc/hosts127.0.0.1 localhost192.168.51.4 hadoop-master 192.168.52.4 hadoop-data1192.168.52.6 hadoop-data2 Here are the startup logs for all three nodes:https://gist.github.com/dwatrous/7241bb804a9be8f9303f https://gist.github.com/dwatrous/bcd85cda23d6eca3a68b https://gist.github.com/dwatrous/922c4f773aded0137fa3 Thanks for your help. On Fri, Sep 25, 2015 at 10:33 AM, Brahma Reddy Battula wrote: Seems DN started in three machines and failed in hadoop-data1(192.168.52.4).. 192.168.51.6 : giving IP as 192.168.51.1...can you please check /etc/hosts file of 192.168.51.6 (might be 192.168.51.1 is configured in /etc/hosts) 192.168.52.4 : datanode startup might be failed ( you can check this node logs) 192.168.51.4 : Datanode starup is success..which is in master node.. Thanks & Regards Brahma Reddy Battula From: Daniel Watrous [dwmaill...@gmail.com] Sent: Friday, September 25, 2015 8:41 PM To: user@hadoop.apache.org Subject: Re: Problem running example (wrong IP address) I'm still stuck on this and posted it to stackoverflow: http://stackoverflow.com/questions/32785256/hadoop-datanode-binds-wrong-ip-address Thanks, Daniel On Fri, Sep 25, 2015 at 8:28 AM, Daniel Watrous wrote: I could really use some help here. As you can see from the output below, the two attached datanodes are identified with a non-existent IP address. Can someone tell me how that gets selected or how to explicitly set it. Also, why are both datanodes shown under the same name/IP? hadoop@hadoop-master:~$ hdfs dfsadmin -report Configured Capacity: 84482326528 (78.68 GB) Present Capacity: 75745546240 (70.54 GB) DFS Remaining: 75744862208 (70.54 GB) DFS Used: 684032 (668 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 - Live datanodes (2): Name: 192.168.51.1:50010 (192.168.51.1) Hostname: hadoop-data1 Decommission Status : Normal Configured Capacity: 42241163264 (39.34 GB) DFS Used: 303104 (296 KB) Non DFS Used: 4302479360 (4.01 GB) DFS Remaining: 37938380800 (35.33 GB) DFS Used%: 0.00% DFS Remaining%: 89.81% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri Sep 25 13:25:37 UTC 2015 Name: 192.168.51.4:50010 (hadoop-master) Hostname: hadoop-master Decommission Status : Normal Configured Capacity: 42241163264 (39.34 GB) DFS Used: 380928 (372 KB) Non DFS Used: 4434300928 (4.13 GB) DFS Remaining: 37806481408 (35.21 GB) DFS Used%: 0.00% DFS Remaining%: 89.50% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Fri Sep 25 13:25:38 UTC 2015 On Thu, Sep 24, 2015 at 5:05 PM, Daniel Watrous wrote: The IP address is clearly wrong, but I'm not sure how it gets set. Can someone tell me how to configure it to choose a valid IP address? On Thu, Sep 24, 2015 at 3:26 PM, Daniel Watrous wrote: I just noticed that both datanodes appear to have chosen that IP address and bound that port for HDFS communication. http://screencast.com/t/OQNbrWFF Any idea why this would be? Is there some way to specify which IP/hostname should be used for that? On Thu, Sep 24, 2015 at 3:11 PM, Daniel Watrous wrote: When I try to run a map reduce example, I get the following error: hadoop@hadoop-master:~$ hadoop jar /usr/local/hadoop/share/hadoop/
Re: Error importing hbase table on new system
Yes, I have tried to read them on another system as well. It worked there. But I don't know if they are HFilev1 or HFilev2 format(any way to check ?? ) This is the first lines from one of the files SEQ1org.apache.hadoop.hbase.io.ImmutableBytesWritable%org.apache.hadoop.hbase.client.Result*org.apache.hadoop.io.compress.DefaultCodec���N�� $��a&t��wb%!10107712083-10152358443612846x���P]�6:�w|pw ��� �$��K 0� Npw�$x�@����s�;�Ɦ���V�^��ݽW���� �қ��<�/��0/�?'/� �/�����7{kG[(������w��OY^I���}9��l��;�TJ�����J�pu���V�ӡm�\E@��V6�oe45U���,�3���Ͻ�w��O���zڼ�/��歇�KȦ/?��Y;�/�������}���룫-�'_�k���q�$��˨�����^ ���i���tH$/��e.J��{S�\��S>Gd���1~p#��o����M �!٠��;c��IkQ �A)|d�i�(Z�f��oPb�j{��x����`�b���cbb`�"�}�HCG��&�JG�%��',*!!�� ������&�_Q��R�2�1��_��~>:�b����w@�B�~Y�H�(�h/FR_+��nX`#� |D����j����f��ƨT��k/颚h��4`+Q#�ⵕ�,Z�80�V:� )Y)4Lq��[�z#���T wrote: > Have you verified that the files to be imported are in HFilev2 format ? > > http://hbase.apache.org/book.html#_hfile_tool > > Cheers > > On Sun, Sep 27, 2015 at 4:47 AM, Håvard Wahl Kongsgård > wrote: >> >> >Is the single node system secure ? >> >> No have not activated, just defaults >> >> the mapred conf. >> >> >> >> >> >> >> >> >> >> >> mapred.job.tracker >> >> rack3:8021 >> >> >> >> >> >> >> >> >> mapred.jobtracker.plugins >> >> org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin >> >> Comma-separated list of jobtracker plug-ins to be >> activated. >> >> >> >> >> >> >> >> jobtracker.thrift.address >> >> 0.0.0.0:9290 >> >> >> >> >> >> >> >>Have you checked hdfs healthiness ? >> >> >> sudo -u hdfs hdfs dfsadmin -report >> >> Configured Capacity: 2876708585472 (2.62 TB) >> >> Present Capacity: 1991514849280 (1.81 TB) >> >> DFS Remaining: 1648230617088 (1.50 TB) >> >> DFS Used: 343284232192 (319.71 GB) >> >> DFS Used%: 17.24% >> >> Under replicated blocks: 52 >> >> Blocks with corrupt replicas: 0 >> >> Missing blocks: 0 >> >> >> - >> >> Datanodes available: 1 (1 total, 0 dead) >> >> >> Live datanodes: >> >> Name: 127.0.0.1:50010 (localhost) >> >> Hostname: rack3 >> >> Decommission Status : Normal >> >> Configured Capacity: 2876708585472 (2.62 TB) >> >> DFS Used: 343284232192 (319.71 GB) >> >> Non DFS Used: 885193736192 (824.40 GB) >> >> DFS Remaining: 1648230617088 (1.50 TB) >> >> DFS Used%: 11.93% >> >> DFS Remaining%: 57.30% >> >> Last contact: Sun Sep 27 13:44:45 CEST 2015 >> >> >> >>To which release of hbase were you importing ? >> >> Hbase 0.94 (CHD 4) >> >> the new one is CHD 5.4 >> >> On Sun, Sep 27, 2015 at 1:32 PM, Ted Yu wrote: >> > Is the single node system secure ? >> > Have you checked hdfs healthiness ? >> > To which release of hbase were you importing ? >> > >> > Thanks >> > >> >> On Sep 27, 2015, at 3:06 AM, Håvard Wahl Kongsgård >> >> wrote: >> >> >> >> Hi, Iam trying to import a old backup to a new smaller system (just >> >> single node, to get the data out) >> >> >> >> when I use >> >> >> >> sudo -u hbase hbase -Dhbase.import.version=0.94 >> >> org.apache.hadoop.hbase.mapreduce.Import crawler >> >> /crawler_hbase/crawler >> >> >> >> I get this error in the tasks . Is this a permission problem? >> >> >> >> >> >> 2015-09-26 23:56:32,995 ERROR >> >> org.apache.hadoop.security.UserGroupInformation: >> >> PriviledgedActionException as:mapred (auth:SIMPLE) >> >> cause:java.io.IOException: keyvalues=NONE read 4096 bytes, should read >> >> 14279 >> >> 2015-09-26 23:56:32,996 WARN org.apache.hadoop.mapred.Child: Error >> >> running child >> >> java.io.IOException: keyvalues=NONE read 4096 bytes, should read 14279 >> >> at >> >> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2221) >> >> at >> >> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74) >> >> at >> >> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483) >> >> at >> >> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76) >> >> at >> >> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85) >> >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139) >> >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) >> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) >> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> at javax.security.auth.Subject.doAs(Subject.java:415) >> >> at >> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) >> >> at org.apache.hadoop.mapred.Child.main(Child.java:262) >> >> 2015-09-26 23:56:33,002 INFO org.apache.hadoop.mapred.Task: Runnning >> >> cleanup for the task >> >> >> >> >> >>
Re: Error importing hbase table on new system
Have you used HFile tool ? The tool would print out that information as part of metadata. Cheers On Sun, Sep 27, 2015 at 9:19 AM, Håvard Wahl Kongsgård < haavard.kongsga...@gmail.com> wrote: > Yes, I have tried to read them on another system as well. It worked > there. But I don't know if they are HFilev1 or HFilev2 format(any way > to check ?? ) > > This is the first lines from one of the files > > SEQ > 1org.apache.hadoop.hbase.io.ImmutableBytesWritable%org.apache.hadoop.hbase.client.Result > *org.apache.hadoop.io.compress.DefaultCodec���N�� > $��a&t ��wb%!10107712083-10152358443612846x��� P ]�6:� w |pw > ��� �$��K 0� Npw�$x�@� ���s�;�Ɦ���V�^��ݽW� �� � > �қ ��<� /��0/ � ?'/ � > � /��� �� 7{kG [(��� ���w�� OY^I ���}9 � �l��;�TJ�� �� �J� ‹ > pu���V� ӡm�\E @ ��V6�oe45U ���,�3 ���Ͻ�w��O���zڼ�/��歇�KȦ/ ?�� > Y;� / ��� �� �� }� ��룫-�'_�k� ��q� $ ��˨� � ���^ > ��� i��� tH$/��e.J��{S �\��S >G d���1~ p#�� o �� ��M > �!٠��;c��I kQ > �A)|d�i�(Z�f��o Pb �j {� �x��� � `�b���cbb`�"�} � > HCG��&�JG�%��',*!!�� > �� � � ��& �_Q��R�2�1��_��~>:� b � ���w @�B� ~Y�H�(�h/FR > _+��nX `#� > |D��� �j���� f ��ƨT��k/ 颚h ��4` +Q#�ⵕ�,Z�80�V:� > )Y)4Lq��[� z#���T > -Håvard > > On Sun, Sep 27, 2015 at 4:06 PM, Ted Yu wrote: > > Have you verified that the files to be imported are in HFilev2 format ? > > > > http://hbase.apache.org/book.html#_hfile_tool > > > > Cheers > > > > On Sun, Sep 27, 2015 at 4:47 AM, Håvard Wahl Kongsgård > > wrote: > >> > >> >Is the single node system secure ? > >> > >> No have not activated, just defaults > >> > >> the mapred conf. > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> mapred.job.tracker > >> > >> rack3:8021 > >> > >> > >> > >> > >> > >> > >> > >> > >> mapred.jobtracker.plugins > >> > >> org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin > >> > >> Comma-separated list of jobtracker plug-ins to be > >> activated. > >> > >> > >> > >> > >> > >> > >> > >> jobtracker.thrift.address > >> > >> 0.0.0.0:9290 > >> > >> > >> > >> > >> > >> > >> >>Have you checked hdfs healthiness ? > >> > >> > >> sudo -u hdfs hdfs dfsadmin -report > >> > >> Configured Capacity: 2876708585472 (2.62 TB) > >> > >> Present Capacity: 1991514849280 (1.81 TB) > >> > >> DFS Remaining: 1648230617088 (1.50 TB) > >> > >> DFS Used: 343284232192 (319.71 GB) > >> > >> DFS Used%: 17.24% > >> > >> Under replicated blocks: 52 > >> > >> Blocks with corrupt replicas: 0 > >> > >> Missing blocks: 0 > >> > >> > >> - > >> > >> Datanodes available: 1 (1 total, 0 dead) > >> > >> > >> Live datanodes: > >> > >> Name: 127.0.0.1:50010 (localhost) > >> > >> Hostname: rack3 > >> > >> Decommission Status : Normal > >> > >> Configured Capacity: 2876708585472 (2.62 TB) > >> > >> DFS Used: 343284232192 (319.71 GB) > >> > >> Non DFS Used: 885193736192 (824.40 GB) > >> > >> DFS Remaining: 1648230617088 (1.50 TB) > >> > >> DFS Used%: 11.93% > >> > >> DFS Remaining%: 57.30% > >> > >> Last contact: Sun Sep 27 13:44:45 CEST 2015 > >> > >> > >> >>To which release of hbase were you importing ? > >> > >> Hbase 0.94 (CHD 4) > >> > >> the new one is CHD 5.4 > >> > >> On Sun, Sep 27, 2015 at 1:32 PM, Ted Yu wrote: > >> > Is the single node system secure ? > >> > Have you checked hdfs healthiness ? > >> > To which release of hbase were you importing ? > >> > > >> > Thanks > >> > > >> >> On Sep 27, 2015, at 3:06 AM, Håvard Wahl Kongsgård > >> >> wrote: > >> >> > >> >> Hi, Iam trying to import a old backup to a new smaller system (just > >> >> single node, to get the data out) > >> >> > >> >> when I use > >> >> > >> >> sudo -u hbase hbase -Dhbase.import.version=0.94 > >> >> org.apache.hadoop.hbase.mapreduce.Import crawler > >> >> /crawler_hbase/crawler > >> >> > >> >> I get this error in the tasks . Is this a permission problem? > >> >> > >> >> > >> >> 2015-09-26 23:56:32,995 ERROR > >> >> org.apache.hadoop.security.UserGroupInformation: > >> >> PriviledgedActionException as:mapred (auth:SIMPLE) > >> >> cause:java.io.IOException: keyvalues=NONE read 4096 bytes, should > read > >> >> 14279 > >> >> 2015-09-26 23:56:32,996 WARN org.apache.hadoop.mapred.Child: Error > >> >> running child > >> >> java.io.IOException: keyvalues=NONE read 4096 bytes, should read > 14279 > >> >> at > >> >> > org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2221) > >> >> at > >> >> > org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74) > >> >> at > >> >> > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483) > >> >> at > >> >> > org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76) > >> >> at > >> >> > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85) > >> >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:
Re: Error importing hbase table on new system
But is not that tool for reading regions not export (dumps) ?? -Håvard On Sun, Sep 27, 2015 at 6:23 PM, Ted Yu wrote: > Have you used HFile tool ? > > The tool would print out that information as part of metadata. > > Cheers > > On Sun, Sep 27, 2015 at 9:19 AM, Håvard Wahl Kongsgård > wrote: >> >> Yes, I have tried to read them on another system as well. It worked >> there. But I don't know if they are HFilev1 or HFilev2 format(any way >> to check ?? ) >> >> This is the first lines from one of the files >> >> SEQ >> 1org.apache.hadoop.hbase.io.ImmutableBytesWritable%org.apache.hadoop.hbase.client.Result >> *org.apache.hadoop.io.compress.DefaultCodec���N�� >> $��a&t ��wb%!10107712083-10152358443612846x��� P ]�6:� w |pw >> ��� �$��K 0� Npw�$x�@� ���s�;�Ɦ���V�^��ݽW� �� � >> �қ ��<� /��0/ � ?'/ � >> � /��� �� 7{kG [(��� ���w�� OY^I ���}9 � �l��;�TJ�� �� �J� ‹ >> pu���V� ӡm�\E @ ��V6�oe45U ���,�3 ���Ͻ�w��O���zڼ�/��歇�KȦ/ ?�� Y;� >> / ��� �� �� }� ��룫-�'_�k� ��q� $ ��˨� � ���^ >> ��� i��� tH$/��e.J��{S �\��S >G d���1~ p#�� o �� ��M >> �!٠��;c��I kQ >> �A)|d�i�(Z�f��o Pb �j {� �x��� � `�b���cbb`�"�} � >> HCG��&�JG�%��',*!!�� >> �� � � ��& �_Q��R�2�1��_��~>:� b � ���w @�B� ~Y�H�(�h/FR >> _+��nX `#� >> |D��� �j���� f ��ƨT��k/ 颚h ��4` +Q#�ⵕ�,Z�80�V:� >> )Y)4Lq��[� z#���T> >> -Håvard >> >> On Sun, Sep 27, 2015 at 4:06 PM, Ted Yu wrote: >> > Have you verified that the files to be imported are in HFilev2 format ? >> > >> > http://hbase.apache.org/book.html#_hfile_tool >> > >> > Cheers >> > >> > On Sun, Sep 27, 2015 at 4:47 AM, Håvard Wahl Kongsgård >> > wrote: >> >> >> >> >Is the single node system secure ? >> >> >> >> No have not activated, just defaults >> >> >> >> the mapred conf. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> mapred.job.tracker >> >> >> >> rack3:8021 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> mapred.jobtracker.plugins >> >> >> >> org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin >> >> >> >> Comma-separated list of jobtracker plug-ins to be >> >> activated. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> jobtracker.thrift.address >> >> >> >> 0.0.0.0:9290 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>Have you checked hdfs healthiness ? >> >> >> >> >> >> sudo -u hdfs hdfs dfsadmin -report >> >> >> >> Configured Capacity: 2876708585472 (2.62 TB) >> >> >> >> Present Capacity: 1991514849280 (1.81 TB) >> >> >> >> DFS Remaining: 1648230617088 (1.50 TB) >> >> >> >> DFS Used: 343284232192 (319.71 GB) >> >> >> >> DFS Used%: 17.24% >> >> >> >> Under replicated blocks: 52 >> >> >> >> Blocks with corrupt replicas: 0 >> >> >> >> Missing blocks: 0 >> >> >> >> >> >> - >> >> >> >> Datanodes available: 1 (1 total, 0 dead) >> >> >> >> >> >> Live datanodes: >> >> >> >> Name: 127.0.0.1:50010 (localhost) >> >> >> >> Hostname: rack3 >> >> >> >> Decommission Status : Normal >> >> >> >> Configured Capacity: 2876708585472 (2.62 TB) >> >> >> >> DFS Used: 343284232192 (319.71 GB) >> >> >> >> Non DFS Used: 885193736192 (824.40 GB) >> >> >> >> DFS Remaining: 1648230617088 (1.50 TB) >> >> >> >> DFS Used%: 11.93% >> >> >> >> DFS Remaining%: 57.30% >> >> >> >> Last contact: Sun Sep 27 13:44:45 CEST 2015 >> >> >> >> >> >> >>To which release of hbase were you importing ? >> >> >> >> Hbase 0.94 (CHD 4) >> >> >> >> the new one is CHD 5.4 >> >> >> >> On Sun, Sep 27, 2015 at 1:32 PM, Ted Yu wrote: >> >> > Is the single node system secure ? >> >> > Have you checked hdfs healthiness ? >> >> > To which release of hbase were you importing ? >> >> > >> >> > Thanks >> >> > >> >> >> On Sep 27, 2015, at 3:06 AM, Håvard Wahl Kongsgård >> >> >> wrote: >> >> >> >> >> >> Hi, Iam trying to import a old backup to a new smaller system (just >> >> >> single node, to get the data out) >> >> >> >> >> >> when I use >> >> >> >> >> >> sudo -u hbase hbase -Dhbase.import.version=0.94 >> >> >> org.apache.hadoop.hbase.mapreduce.Import crawler >> >> >> /crawler_hbase/crawler >> >> >> >> >> >> I get this error in the tasks . Is this a permission problem? >> >> >> >> >> >> >> >> >> 2015-09-26 23:56:32,995 ERROR >> >> >> org.apache.hadoop.security.UserGroupInformation: >> >> >> PriviledgedActionException as:mapred (auth:SIMPLE) >> >> >> cause:java.io.IOException: keyvalues=NONE read 4096 bytes, should >> >> >> read >> >> >> 14279 >> >> >> 2015-09-26 23:56:32,996 WARN org.apache.hadoop.mapred.Child: Error >> >> >> running child >> >> >> java.io.IOException: keyvalues=NONE read 4096 bytes, should read >> >> >> 14279 >> >> >> at >> >> >> >> >> >> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2221) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReade
RE: Why would ApplicationManager request RAM more that defaut 1GB?
Hi Ilya, I think that property is of less significance, its only to confirm wrt behavior wrt virtual memory. But the imp one is, can we get the snapshot of the heap (using the command shared earlier), from it roughly we can determine which object is hogging the memory. + Naga From: Ilya Karpov [i.kar...@cleverdata.ru] Sent: Friday, September 25, 2015 14:34 To: user@hadoop.apache.org Subject: Re: Why would ApplicationManager request RAM more that defaut 1GB? Hi Manoj & Naga, I’m surprised but there is no such a property in CHD conf files (greped all *.xml in OSes where yarn lives!) I think that this property is set by Cloudera: http://image.slidesharecdn.com/yarnsaboutyarn-kathleenting112114-141125155911-conversion-gate01/95/yarns-about-yarn-28-638.jpg?cb=1416931543 (we use chd 5.4.5) 25 сент. 2015 г., в 10:19, Naganarasimha Garla mailto:naganarasimha...@gmail.com>> написал(а): Hi Manoj & Ilya, >From the logs 2015-09-21 22:50:34,018 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=13982,containerID=container_1442402147223_0165_01_01] is running beyond physical memory limits. This indicates that its increase in Physical memory limits and not virtual limits due to which container was killed and also probability that Container Monitor would wait till vmem is 3.4 GB when limit is 2.1 GB to kill it is less. Vmem includes overall memory limits including the files opened etc ... but seems like virtual mem check in your setup seems to be disabled. please check for configuration as mentioned by Manoj "yarn.nodemanager.vmem-check-enabled " to cross verify On Fri, Sep 25, 2015 at 12:15 PM, Ilya Karpov mailto:i.kar...@cleverdata.ru>> wrote: Hello, Manoj the actual question is why this happens 24 сент. 2015 г., в 20:39, manoj mailto:manojm@gmail.com>> написал(а): Hello IIya, Looks like the Vmem usage is going above the above 2.1 of Pmem times thats why the container is getting killed, 1.0 GB of 1 GB physical memory used; 3.4 GB of 2.1 GB virtual memory used By default Vmem is set to 2.1 times of the Pmem. Looks like your job is taking 3.4GB! You can change the ratio by setting in Yarn-site.xml: yarn.nodemanager.vmem-pmem-ratio You can optionally disable this check by setting following to false: yarn.nodemanager.vmem-check-enabled Thanks, -Manoj On Wed, Sep 23, 2015 at 12:36 AM, Ilya Karpov mailto:i.kar...@cleverdata.ru>> wrote: Great thanks for your reply! >1. Which version of Hadoop/ YARN ? Hadoop(command: hadoop version): Hadoop 2.6.0-cdh5.4.5 Subversion http://github.com/cloudera/hadoop -r ab14c89fe25e9fb3f9de4fb852c21365b7c5608b Compiled by jenkins on 2015-08-12T21:11Z Compiled with protoc 2.5.0 >From source with checksum d31cb7e46b8602edaf68d335b785ab This command was run using /opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/jars/hadoop-common-2.6.0-cdh5.4.5.jar Yarn (command: yarn version) prints exactly the same. >2. From the logs is it getting killed due to over usage of Vmem or Physical >memory ? Because of over usage of Physical memory. Last seconds of life: 2015-09-21 22:50:34,017 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 13982 for container-id container_1442402147223_0165_01_01: 1.0 GB of 1 GB physical memory used; 3.4 GB of 2.1 GB virtual memory used 2015-09-21 22:50:34,017 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Process tree for container: container_1442402147223_0165_01_01 has processes older than 1 iteration running over the configured limit. Limit=1073741824, current usage = 1074352128 2015-09-21 22:50:34,018 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=13982,containerID=container_1442402147223_0165_01_01] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 3.4 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_1442402147223_0165_01_01 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 13994 13982 13982 13982 (java) 4285 714 3602911232 261607 /opt/jdk1.8.0_60/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/contai ner/application_1442402147223_0165/container_1442402147223_0165_01_01 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Djava.net.preferIPv4Stack=true -Xmx825955249 org.apache.had oop.mapreduce.v2.app.MRAppMaster |- 13982 13980 13982 13982 (bash) 0 0 14020608 686 /bin/bash -c /opt/jdk1.8.0_60/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1442402147223_0165/container_