Re: Need help to understand hadoop.tmp.dir

2014-03-03 Thread Chengwei Yang 
On Mon, Mar 03, 2014 at 11:56:08AM +0530, shashwat shriparv wrote:
> Ya its always better to change the temp dir path in hadoop, as it will prevent
> deletion of file while the server reboots.

Thanks, so is there anyway to recovery from this state? Or I have to format
namenode again?

--
Thanks,
Chengwei

> 
> 
> Warm Regards_∞_
> Shashwat Shriparv
> http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://twitter.com/
> shriparvhttps://www.facebook.com/shriparvhttp://google.com/+ShashwatShriparv
> http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> SWXSTW3DVSDTF2HHSRM47AV6DI/
> 
> 
> 
> On Mon, Mar 3, 2014 at 11:52 AM, Chengwei Yang 
> wrote:
> 
> On Mon, Mar 03, 2014 at 11:25:59AM +0530, shashwat shriparv wrote:
> > You can use any directory you like beside permissions are right.
> 
> I mean if it's better if we change the default hadoop.tmp.dir? Because it
> can not work cross reboot in default Linux environment.
> 
> --
> Thanks,
> Chengwei
> 
> >
> >
> > Warm Regards_∞_
> > Shashwat Shriparv
> > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> twitter.com/
> > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> +ShashwatShriparv
> > http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> > SWXSTW3DVSDTF2HHSRM47AV6DI/
> >
> >
> >
> > On Mon, Mar 3, 2014 at 11:07 AM, Chengwei Yang <
> chengwei.yang...@gmail.com>
> > wrote:
> >
> >     Hi List,
> >
> >     I'm confusing by hadoop.tmp.dir currently because its default value
> >     "/tmp/hadoop-${user.name}" always means a directory in tmpfs in
> Linux.
> >     So after the name node machine reboot, it gone away and then name
> node
> >     fail to start.
> >
> >     I found this was reported here.
> >     
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201205.mbox
> /
> >     %3cbay148-w22bf95c5fbe2c40bf7cd9f86...@phx.gbl%3E
> >
> >     As I found from http://hadoop.apache.org/docs/r2.3.0/, there are a
> lot
> >     properties are based on hadoop.tmp.dir, like
> >     dfs.namenode.name.dir   file://${hadoop.tmp.dir}/dfs/name
> >
> >     I'm wondering, if we can set the default value of hadoop.tmp.dir to
> >     a non-tmpfs direcotry if it doesn't work at all by using a real 
> tmpfs
> >     directory?
> >
> >     --
> >     Thanks,
> >     Chengwei
> >
> >
> 
>

Re: Need help to understand hadoop.tmp.dir

2014-03-03 Thread shashwat shriparv 
NO need to format just change the value and restart the cluster;


*Warm Regards_**∞_*
* Shashwat Shriparv*
 [image: 
http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9][image:
https://twitter.com/shriparv] [image:
https://www.facebook.com/shriparv] [image:
http://google.com/+ShashwatShriparv]
[image:
http://www.youtube.com/user/sShriparv/videos][image:
http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/] 



On Mon, Mar 3, 2014 at 1:55 PM, Chengwei Yang wrote:

> On Mon, Mar 03, 2014 at 11:56:08AM +0530, shashwat shriparv wrote:
> > Ya its always better to change the temp dir path in hadoop, as it will
> prevent
> > deletion of file while the server reboots.
>
> Thanks, so is there anyway to recovery from this state? Or I have to format
> namenode again?
>
> --
> Thanks,
> Chengwei
>
> >
> >
> > Warm Regards_∞_
> > Shashwat Shriparv
> >
> http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://twitter.com/
> > shriparvhttps://
> www.facebook.com/shriparvhttp://google.com/+ShashwatShriparv
> > http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> > SWXSTW3DVSDTF2HHSRM47AV6DI/
> >
> >
> >
> > On Mon, Mar 3, 2014 at 11:52 AM, Chengwei Yang <
> chengwei.yang...@gmail.com>
> > wrote:
> >
> > On Mon, Mar 03, 2014 at 11:25:59AM +0530, shashwat shriparv wrote:
> > > You can use any directory you like beside permissions are right.
> >
> > I mean if it's better if we change the default hadoop.tmp.dir?
> Because it
> > can not work cross reboot in default Linux environment.
> >
> > --
> > Thanks,
> > Chengwei
> >
> > >
> > >
> > > Warm Regards_∞_
> > > Shashwat Shriparv
> > > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> > twitter.com/
> > > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> > +ShashwatShriparv
> > >
> http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> > > SWXSTW3DVSDTF2HHSRM47AV6DI/
> > >
> > >
> > >
> > > On Mon, Mar 3, 2014 at 11:07 AM, Chengwei Yang <
> > chengwei.yang...@gmail.com>
> > > wrote:
> > >
> > > Hi List,
> > >
> > > I'm confusing by hadoop.tmp.dir currently because its default
> value
> > > "/tmp/hadoop-${user.name}" always means a directory in tmpfs
> in
> > Linux.
> > > So after the name node machine reboot, it gone away and then
> name
> > node
> > > fail to start.
> > >
> > > I found this was reported here.
> > >
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201205.mbox
> > /
> > > %3cbay148-w22bf95c5fbe2c40bf7cd9f86...@phx.gbl%3E
> > >
> > > As I found from http://hadoop.apache.org/docs/r2.3.0/, there
> are a
> > lot
> > > properties are based on hadoop.tmp.dir, like
> > > dfs.namenode.name.dir   file://${hadoop.tmp.dir}/dfs/name
> > >
> > > I'm wondering, if we can set the default value of
> hadoop.tmp.dir to
> > > a non-tmpfs direcotry if it doesn't work at all by using a
> real tmpfs
> > > directory?
> > >
> > > --
> > > Thanks,
> > > Chengwei
> > >
> > >
> >
> >
>

Re: Need help to understand hadoop.tmp.dir

2014-03-03 Thread Chengwei Yang 
On Mon, Mar 03, 2014 at 01:57:49PM +0530, shashwat shriparv wrote:
> NO need to format just change the value and restart the cluster;

Hmm, seems it doesn't work for me, if the only need to do is to change
to another directory, then why it can not re-init the directory in /tmp
just as another directory?

If I changed to another directory, a new directory, the same error
happen.

--
Thanks,
Chengwei

> 
> 
> Warm Regards_∞_
> Shashwat Shriparv
> http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://twitter.com/
> shriparvhttps://www.facebook.com/shriparvhttp://google.com/+ShashwatShriparv
> http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> SWXSTW3DVSDTF2HHSRM47AV6DI/
> 
> 
> 
> On Mon, Mar 3, 2014 at 1:55 PM, Chengwei Yang 
> wrote:
> 
> On Mon, Mar 03, 2014 at 11:56:08AM +0530, shashwat shriparv wrote:
> > Ya its always better to change the temp dir path in hadoop, as it will
> prevent
> > deletion of file while the server reboots.
> 
> Thanks, so is there anyway to recovery from this state? Or I have to 
> format
> namenode again?
> 
> --
> Thanks,
> Chengwei
> 
> >
> >
> > Warm Regards_∞_
> > Shashwat Shriparv
> > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> twitter.com/
> > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> +ShashwatShriparv
> > http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> > SWXSTW3DVSDTF2HHSRM47AV6DI/
> >
> >
> >
> > On Mon, Mar 3, 2014 at 11:52 AM, Chengwei Yang <
> chengwei.yang...@gmail.com>
> > wrote:
> >
> >     On Mon, Mar 03, 2014 at 11:25:59AM +0530, shashwat shriparv wrote:
> >     > You can use any directory you like beside permissions are right.
> >
> >     I mean if it's better if we change the default hadoop.tmp.dir?
> Because it
> >     can not work cross reboot in default Linux environment.
> >
> >     --
> >     Thanks,
> >     Chengwei
> >
> >     >
> >     >
> >     > Warm Regards_∞_
> >     > Shashwat Shriparv
> >     > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> >     twitter.com/
> >     > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> >     +ShashwatShriparv
> >     > http://www.youtube.com/user/sShriparv/videoshttp://
> profile.yahoo.com/
> >     > SWXSTW3DVSDTF2HHSRM47AV6DI/
> >     >
> >     >
> >     >
> >     > On Mon, Mar 3, 2014 at 11:07 AM, Chengwei Yang <
> >     chengwei.yang...@gmail.com>
> >     > wrote:
> >     >
> >     >     Hi List,
> >     >
> >     >     I'm confusing by hadoop.tmp.dir currently because its default
> value
> >     >     "/tmp/hadoop-${user.name}" always means a directory in tmpfs 
> in
> >     Linux.
> >     >     So after the name node machine reboot, it gone away and then
> name
> >     node
> >     >     fail to start.
> >     >
> >     >     I found this was reported here.
> >     >     http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/
> 201205.mbox
> >     /
> >     >     %3cbay148-w22bf95c5fbe2c40bf7cd9f86...@phx.gbl%3E
> >     >
> >     >     As I found from http://hadoop.apache.org/docs/r2.3.0/, there
> are a
> >     lot
> >     >     properties are based on hadoop.tmp.dir, like
> >     >     dfs.namenode.name.dir   file://${hadoop.tmp.dir}/dfs/name
> >     >
> >     >     I'm wondering, if we can set the default value of
> hadoop.tmp.dir to
> >     >     a non-tmpfs direcotry if it doesn't work at all by using a 
> real
> tmpfs
> >     >     directory?
> >     >
> >     >     --
> >     >     Thanks,
> >     >     Chengwei
> >     >
> >     >
> >
> >
> 
>

Dhruba is in workshop with client

2014-03-03 Thread Dhruba Bhattacharjee1 

I will be out of the office starting  03/03/2014 and will not return until
03/06/2014.

I am attending workshops with client till 6th March, there might be delays
in response.

Warm Regards
Dhrubo

ResourceManager crash on deleted NM node back from the dead

2014-03-03 Thread John Lilley 
We had a DN/NM node that went offline for a while and been removed from the 
cluster via Ambari without decommissioning (because it was offline).
When the node came back up, its NM attempted connection to the RM.
Later the RM failed with this exception (dokken is the errant node):

2014-03-03 05:53:28,721 FATAL resourcemanager.ResourceManager 
(ResourceManager.java:run(449)) - Error in handling event type NODE_UPDATE to 
the scheduler
java.lang.IllegalArgumentException: java.net.UnknownHostException: 
dokken.office.datalever.com
at 
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
at 
org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerToken(BuilderUtils.java:247)
at 
org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager.createContainerToken(RMContainerTokenSecretManager.java:195)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainerToken(LeafQueue.java:1297)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1345)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1211)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1170)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:871)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:645)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:559)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:690)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:734)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:440)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.UnknownHostException: dokken.office.datalever.com
... 15 more
2014-03-03 05:53:28,723 INFO  resourcemanager.ResourceManager 
(ResourceManager.java:run(453)) - Exiting, bbye..

I can see our error in how this node was removed while not being truly dead, 
but I think it shouldn't crash the RM.
Forgive my ignorance ... do you know if this is a known problem?

FYI:
# hadoop version
Hadoop 2.2.0.2.0.6.0-76
Subversion g...@github.com:hortonworks/hadoop.git -r 
8656b1cfad13b03b29e98cad042626205e7a1c86
Compiled by jenkins on 2013-10-18T00:19Z
Compiled with protoc 2.5.0
>From source with checksum d23ee1d271c6ac5bd27de664146be2
This command was run using /usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-76.jar

Thanks
John

Re: Need help to understand hadoop.tmp.dir

2014-03-03 Thread JCAD Cell 1 
With the services stopped you would change the setting in core-site.xml:
  
hadoop.tmp.dir
/var/hadoop/tmp
  

Then move your /tmp/hadoop folder over to the new location:
mv /tmp/hadoop /var/hadoop/tmp



On Mon, Mar 3, 2014 at 5:55 AM, Chengwei Yang wrote:

> On Mon, Mar 03, 2014 at 01:57:49PM +0530, shashwat shriparv wrote:
> > NO need to format just change the value and restart the cluster;
>
> Hmm, seems it doesn't work for me, if the only need to do is to change
> to another directory, then why it can not re-init the directory in /tmp
> just as another directory?
>
> If I changed to another directory, a new directory, the same error
> happen.
>
> --
> Thanks,
> Chengwei
>
> >
> >
> > Warm Regards_∞_
> > Shashwat Shriparv
> >
> http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://twitter.com/
> > shriparvhttps://
> www.facebook.com/shriparvhttp://google.com/+ShashwatShriparv
> > http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> > SWXSTW3DVSDTF2HHSRM47AV6DI/
> >
> >
> >
> > On Mon, Mar 3, 2014 at 1:55 PM, Chengwei Yang <
> chengwei.yang...@gmail.com>
> > wrote:
> >
> > On Mon, Mar 03, 2014 at 11:56:08AM +0530, shashwat shriparv wrote:
> > > Ya its always better to change the temp dir path in hadoop, as it
> will
> > prevent
> > > deletion of file while the server reboots.
> >
> > Thanks, so is there anyway to recovery from this state? Or I have to
> format
> > namenode again?
> >
> > --
> > Thanks,
> > Chengwei
> >
> > >
> > >
> > > Warm Regards_∞_
> > > Shashwat Shriparv
> > > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> > twitter.com/
> > > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> > +ShashwatShriparv
> > >
> http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> > > SWXSTW3DVSDTF2HHSRM47AV6DI/
> > >
> > >
> > >
> > > On Mon, Mar 3, 2014 at 11:52 AM, Chengwei Yang <
> > chengwei.yang...@gmail.com>
> > > wrote:
> > >
> > > On Mon, Mar 03, 2014 at 11:25:59AM +0530, shashwat shriparv
> wrote:
> > > > You can use any directory you like beside permissions are
> right.
> > >
> > > I mean if it's better if we change the default hadoop.tmp.dir?
> > Because it
> > > can not work cross reboot in default Linux environment.
> > >
> > > --
> > > Thanks,
> > > Chengwei
> > >
> > > >
> > > >
> > > > Warm Regards_∞_
> > > > Shashwat Shriparv
> > > >
> http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> > > twitter.com/
> > > > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> > > +ShashwatShriparv
> > > > http://www.youtube.com/user/sShriparv/videoshttp://
> > profile.yahoo.com/
> > > > SWXSTW3DVSDTF2HHSRM47AV6DI/
> > > >
> > > >
> > > >
> > > > On Mon, Mar 3, 2014 at 11:07 AM, Chengwei Yang <
> > > chengwei.yang...@gmail.com>
> > > > wrote:
> > > >
> > > > Hi List,
> > > >
> > > > I'm confusing by hadoop.tmp.dir currently because its
> default
> > value
> > > > "/tmp/hadoop-${user.name}" always means a directory in
> tmpfs in
> > > Linux.
> > > > So after the name node machine reboot, it gone away and
> then
> > name
> > > node
> > > > fail to start.
> > > >
> > > > I found this was reported here.
> > > >
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/
> > 201205.mbox
> > > /
> > > > %3cbay148-w22bf95c5fbe2c40bf7cd9f86...@phx.gbl%3E
> > > >
> > > > As I found from http://hadoop.apache.org/docs/r2.3.0/,
> there
> > are a
> > > lot
> > > > properties are based on hadoop.tmp.dir, like
> > > > dfs.namenode.name.dir   file://${hadoop.tmp.dir}/dfs/name
> > > >
> > > > I'm wondering, if we can set the default value of
> > hadoop.tmp.dir to
> > > > a non-tmpfs direcotry if it doesn't work at all by using
> a real
> > tmpfs
> > > > directory?
> > > >
> > > > --
> > > > Thanks,
> > > > Chengwei
> > > >
> > > >
> > >
> > >
> >
> >
>

Re: Drawbacks of Hadoop Pipes

2014-03-03 Thread Basu,Indrashish 


Hello,

Anyone can help regarding the below query.

Regards,
Indrashish

On Sat, 01 Mar 2014 13:52:11 -0500, Basu,Indrashish wrote:

Hello,

I am trying to execute a CUDA benchmark in a Hadoop Framework and
using Hadoop Pipes for invoking the CUDA code which is written in a
C++ interface from the Hadoop Framework. I am just a bit interested 
in

knowing what can be the drawbacks of using Hadoop Pipes for this and
whether the implementation of Hadoop Streaming and JNI interface will
be a better choice. I am a bit unclear on this, so if anyone can 
throw

some light on this and clarify.

Regards,
Indrashish


--
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida

[hadoop] AvroMultipleOutputs org.apache.avro.file.DataFileWriter$AppendWriteException

2014-03-03 Thread John Pauley 
This is cross posted to avro-user list 
(http://mail-archives.apache.org/mod_mbox/avro-user/201402.mbox/%3ccf3612f6.94d2%25john.pau...@threattrack.com%3e).

Hello all,

I’m having an issue using AvroMultipleOutputs in a map/reduce job.  The issue 
occurs when using a schema that has a union of null and a fixed (among other 
complex types), default to null, and it is not null.  Please find the full 
stack trace below and a sample map/reduce job that generates an Avro container 
file and uses that for the m/r input.  Note that I can serialize/deserialize 
without issue using GenericDatumWriter/GenericDatumReader outside of hadoop…  
Any insight would be helpful.

Stack trace:
java.lang.Exception: org.apache.avro.file.DataFileWriter$AppendWriteException: 
java.lang.NullPointerException: in com.foo.bar.simple_schema in union null of 
union in field baz of com.foo.bar.simple_schema
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: 
java.lang.NullPointerException: in com.foo.bar.simple_schema in union null of 
union in field baz of com.foo.bar.simple_schema
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296)
at 
org.apache.avro.mapreduce.AvroKeyRecordWriter.write(AvroKeyRecordWriter.java:77)
at 
org.apache.avro.mapreduce.AvroKeyRecordWriter.write(AvroKeyRecordWriter.java:39)
at 
org.apache.avro.mapreduce.AvroMultipleOutputs.write(AvroMultipleOutputs.java:400)
at 
org.apache.avro.mapreduce.AvroMultipleOutputs.write(AvroMultipleOutputs.java:378)
at 
com.tts.ox.mapreduce.example.avro.AvroContainerFileDriver$SampleMapper.map(AvroContainerFileDriver.java:78)
at 
com.tts.ox.mapreduce.example.avro.AvroContainerFileDriver$SampleMapper.map(AvroContainerFileDriver.java:62)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at 
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: java.lang.NullPointerException: in com.foo.bar.simple_schema in 
union null of union in field baz of com.foo.bar.simple_schema
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
... 16 more
Caused by: java.lang.NullPointerException
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:457)
at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:189)
at org.apache.avro.reflect.ReflectData.isRecord(ReflectData.java:167)
at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:608)
at org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:265)
at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:597)
at 
org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
at 
org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
at 
org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:175)
at 
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)

Sample m/r job:

package com.tts.ox.mapreduce.example.avro;

import org.apache.avro.Schema;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.generic.GenericRecordBuilder;
import org.apache.avro.io.DatumWriter;
import org.apache.avro.mapred.AvroKey;
import org.apache.avro.mapreduce.AvroJob;
import org.apache.avro.mapreduce.AvroKeyInputFormat;
import org.apache.avro.mapreduce.AvroKeyOutputFormat;
import org.apache.avro.mapreduce.AvroMultipleOutputs;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.out

class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final method getUnknownFields

2014-03-03 Thread Margusja 

Hi

I even don't know what information to provide but my container log is:

2014-03-03 17:36:05,311 FATAL [main] 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.VerifyError: class 
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final 
method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2493)
at java.lang.Class.getConstructor0(Class.java:2803)
at java.lang.Class.getConstructor(Class.java:1718)
at 
org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java:62)
at org.apache.hadoop.yarn.util.Records.newRecord(Records.java:36)
at 
org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49)
at 
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
at 
org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1343)


Where to start digging?

--
Tervitades, Margus (Margusja) Roo
+372 51 48 780
http://margus.roo.ee
http://ee.linkedin.com/in/margusroo
skype: margusja
ldapsearch -x -h ldap.sk.ee -b c=EE "(serialNumber=37303140314)"
-BEGIN PUBLIC KEY-
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
BjM8j36yJvoBVsfOHQIDAQAB
-END PUBLIC KEY-

Re: class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final method getUnknownFields

2014-03-03 Thread Ted Yu 
Can you tell us the hadoop release you're using ?

Seems there is inconsistency in protobuf library.


On Mon, Mar 3, 2014 at 8:01 AM, Margusja  wrote:

> Hi
>
> I even don't know what information to provide but my container log is:
>
> 2014-03-03 17:36:05,311 FATAL [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
> Error starting MRAppMaster
> java.lang.VerifyError: class 
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto
> overrides final method getUnknownFields.()Lcom/google/protobuf/
> UnknownFieldSet;
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(
> SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2493)
> at java.lang.Class.getConstructor0(Class.java:2803)
> at java.lang.Class.getConstructor(Class.java:1718)
> at org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.
> newRecordInstance(RecordFactoryPBImpl.java:62)
> at org.apache.hadoop.yarn.util.Records.newRecord(Records.java:36)
> at org.apache.hadoop.yarn.api.records.ApplicationId.
> newInstance(ApplicationId.java:49)
> at org.apache.hadoop.yarn.util.ConverterUtils.
> toApplicationAttemptId(ConverterUtils.java:137)
> at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(
> ConverterUtils.java:177)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(
> MRAppMaster.java:1343)
>
>
> Where to start digging?
>
> --
> Tervitades, Margus (Margusja) Roo
> +372 51 48 780
> http://margus.roo.ee
> http://ee.linkedin.com/in/margusroo
> skype: margusja
> ldapsearch -x -h ldap.sk.ee -b c=EE "(serialNumber=37303140314)"
> -BEGIN PUBLIC KEY-
> MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE
> 5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl
> RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa
> BjM8j36yJvoBVsfOHQIDAQAB
> -END PUBLIC KEY-
>
>

Re: class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final method getUnknownFields

2014-03-03 Thread Margusja 

Hi

2.2.0 and 2.3.0 gave me the same container log.

A little bit more details.
I'll try to use external java client who submits job.
some lines from maven pom.xml file:

  org.apache.hadoop
  hadoop-client
  2.3.0


org.apache.hadoop
hadoop-core
1.2.1


lines from external client:
...
2014-03-03 17:36:01 INFO  FileInputFormat:287 - Total input paths to 
process : 1

2014-03-03 17:36:02 INFO  JobSubmitter:396 - number of splits:1
2014-03-03 17:36:03 INFO  JobSubmitter:479 - Submitting tokens for job: 
job_1393848686226_0018
2014-03-03 17:36:04 INFO  YarnClientImpl:166 - Submitted application 
application_1393848686226_0018
2014-03-03 17:36:04 INFO  Job:1289 - The url to track the job: 
http://vm38.dbweb.ee:8088/proxy/application_1393848686226_0018/

2014-03-03 17:36:04 INFO  Job:1334 - Running job: job_1393848686226_0018
2014-03-03 17:36:10 INFO  Job:1355 - Job job_1393848686226_0018 running 
in uber mode : false

2014-03-03 17:36:10 INFO  Job:1362 -  map 0% reduce 0%
2014-03-03 17:36:10 INFO  Job:1375 - Job job_1393848686226_0018 failed 
with state FAILED due to: Application application_1393848686226_0018 
failed 2 times due to AM Container for 
appattempt_1393848686226_0018_02 exited with  exitCode: 1 due to: 
Exception from container-launch:

org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)
...

Lines from namenode:
...
14/03/03 19:12:42 INFO namenode.FSEditLog: Number of transactions: 900 
Total time for transactions(ms): 69 Number of transactions batched in 
Syncs: 0 Number of syncs: 542 SyncTimes(ms): 9783
14/03/03 19:12:42 INFO BlockStateChange: BLOCK* addToInvalidates: 
blk_1073742050_1226 90.190.106.33:50010
14/03/03 19:12:42 INFO hdfs.StateChange: BLOCK* allocateBlock: 
/user/hduser/input/data666.noheader.data. 
BP-802201089-90.190.106.33-1393506052071 
blk_1073742056_1232{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]}
14/03/03 19:12:44 INFO hdfs.StateChange: BLOCK* InvalidateBlocks: ask 
90.190.106.33:50010 to delete [blk_1073742050_1226]
14/03/03 19:12:53 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap 
updated: 90.190.106.33:50010 is added to 
blk_1073742056_1232{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]} size 0
14/03/03 19:12:53 INFO hdfs.StateChange: DIR* completeFile: 
/user/hduser/input/data666.noheader.data is closed by 
DFSClient_NONMAPREDUCE_-915999412_15
14/03/03 19:12:54 INFO BlockStateChange: BLOCK* addToInvalidates: 
blk_1073742051_1227 90.190.106.33:50010
14/03/03 19:12:54 INFO hdfs.StateChange: BLOCK* allocateBlock: 
/user/hduser/input/data666.noheader.data.info. 
BP-802201089-90.190.106.33-1393506052071 
blk_1073742057_1233{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]}
14/03/03 19:12:54 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap 
updated: 90.190.106.33:50010 is added to 
blk_1073742057_1233{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]} size 0
14/03/03 19:12:54 INFO hdfs.StateChange: DIR* completeFile: 
/user/hduser/input/data666.noheader.data.info is closed by 
DFSClient_NONMAPREDUCE_-915999412_15
14/03/03 19:12:55 INFO hdfs.StateChange: BLOCK* allocateBlock: 
/user/hduser/.staging/job_1393848686226_0019/job.jar. 
BP-802201089-90.190.106.33-1393506052071 
blk_1073742058_1234{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]}
14/03/03 19:12:56 INFO hdfs.StateChange: BLOCK* InvalidateBlocks: ask 
90.190.106.33:50010 to delete [blk_1073742051_1227]
14/03/03 19:13:12 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap 
updated: 90.190.106.33:50010 is added to 
blk_1073742058_1234{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]} size 0
14/03/03 19:13:12 INFO hdfs.StateChange: DIR* completeFile: 
/user/hduser/.staging/job_1393848686226_0019/job.jar

RE: Huge disk IO on only one disk

2014-03-03 Thread Siddharth Tiwari 
Thanks Brahma,
That answers my question.

**

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of 
God.” 

"Maybe other people will try to limit me but I don't limit myself"


From: brahmareddy.batt...@huawei.com
To: user@hadoop.apache.org
Subject: RE: Huge disk IO on only one disk
Date: Mon, 3 Mar 2014 06:51:30 +








 

 

What should be the standard around setting up the hadoop.tmp.dir parameter.

 

 As I know hadoop.tmp.dir   will be used for follow properites, If you 
 are configuring following properties,then you no need to configure 
 this one..

 

 

MapReduce:

 





mapreduce.cluster.local.dir
${hadoop.tmp.dir}/mapred/local
The local directory where MapReduce stores intermediate data files. May be a 
comma-separated list of directories on different devices in order to spread 
disk i/o. Directories that do not exist are ignored.



mapreduce.jobtracker.system.dir
${hadoop.tmp.dir}/mapred/system
The directory where MapReduce stores control files. 


mapreduce.jobtracker.staging.root.dir
${hadoop.tmp.dir}/mapred/staging
The root of the staging area for users' job files In practice, this should be 
the directory where users' home directories are located (usually /user)



mapreduce.cluster.temp.dir
${hadoop.tmp.dir}/mapred/temp
A shared directory for temporary files. 






 

Yarn : 

 





yarn.nodemanager.local-dirs
${hadoop.tmp.dir}/nm-local-dir
List of directories to store localized files in. An application's localized 
file directory will be found in: 
${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. 
Individual containers' work directories, called
 container_${contid}, will be subdirectories of this. 





 

 

HDFS :

 





dfs.namenode.name.dir
file://${hadoop.tmp.dir}/dfs/name
Determines where on the local filesystem the DFS name node should store the 
name table(fsimage). If this is a comma-delimited list of directories then the 
name table is replicated in all of the directories, for redundancy.






 





dfs.datanode.data.dir
file://${hadoop.tmp.dir}/dfs/data
Determines where on the local filesystem an DFS data node should store its 
blocks. If this is a comma-delimited list of directories, then data will be 
stored in all named directories, typically on different devices. Directories 
that do not exist
 are ignored. 





 





dfs.namenode.checkpoint.dir
file://${hadoop.tmp.dir}/dfs/namesecondary
Determines where on the local filesystem the DFS secondary name node should 
store the temporary images to merge. If this is a comma-delimited list of 
directories then the image is replicated in all of the directories for 
redundancy.







 

 

 

 

Thanks & Regards

 

Brahma Reddy Battula

 





From: Siddharth Tiwari [siddharth.tiw...@live.com]

Sent: Monday, March 03, 2014 11:20 AM

To: USers Hadoop

Subject: RE: Huge disk IO on only one disk






Hi Brahma,



No I havnt, I have put comma separated list of disks here dfs.datanode.data.dir 
. Have
 put disk5 for hadoop.tmp.dir. My Q is, should we set up hadoop.tmp.dir or not 
? if yes what should be standards around.




**

Cheers !!!

Siddharth
Tiwari

Have a refreshing day !!!

"Every duty is holy, and devotion to duty is the highest form of worship of 
God.”


"Maybe other people will try to limit me but I don't limit myself"







From: brahmareddy.batt...@huawei.com

To: user@hadoop.apache.org

Subject: RE: Huge disk IO on only one disk

Date: Mon, 3 Mar 2014 05:14:34 +







 

Seems to be you had started cluster with default values for the following two 
properties and configured for only hadoop.tmp.dir .

 

dfs.datanode.data.dir --->  file://${hadoop.tmp.dir}/dfs/data (Default value)

 

Determines where on the local filesystem an DFS data node should store its 
blocks. If this is a comma-delimited list of directories, then data will be 
stored in all named directories, typically on different devices

 

yarn.nodemanager.local-dirs -->  ${hadoop.tmp.dir}/nm-local-dir (Default value)

 

>>To store localized files, It's like inetermediate files

 

 

Please configure above two values as muliple dir's..



 

 

Thanks & Regards 

Brahma Reddy Battula

 





From: Siddharth Tiwari [siddharth.tiw...@live.com]

Sent: Monday, March 03, 2014 5:58 AM

To: USers Hadoop

Subject: Huge disk IO on only one disk






Hi Team,



I have 10 disks over which I am running my HDFS. Out of this on disk5 I have my 
hadoop.tmp.dir configured. I see that on this disk I have huge IO when I run my 
jobs compared to other disks. Can you guide my to the standards
 to follow so that this IO can be distributed across to other disks as well. 
What should be the standard around setting up the hadoop.tmp.dir parameter. 
Any help would be highly appreciated. below is IO while I am running a huge job.








Device:tps

Not information in Job History UI

2014-03-03 Thread SF Hadoop 
Hadoop 2.2.0
CentOS 6.4
Viewing UI in various browsers.

I am having a problem where no information is visible in my Job History UI.
 I run test jobs, they complete without error, but no information ever
populates the nodemanager or jobhistory server UI.

Also, there are no job logs generated on the server that is running the job.

I have the following settings configured:
yarn.nodemanager.local-dirs
yarn.nodemanager.log-dirs
yarn.log.server.url

...plus the basic yarn log dir.  I get output in regards to the daemons but
very little in regards to the job.  All I get that refers to the jobhistory
server is the following (so it appears to be functioning properly):

2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 19888
2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
to /tmp/Jetty_server_19888_jobhistoryv7gnnv/webapp
2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
SelectChannelConnector@server:19888
2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
/jobhistory started at 19888
2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
Registered webapp guice modules

I have a feeling this is a misconfiguration but I cannot figure out what
setting is missing or wrong.

Other than not being able to see any of the jobs in the UIs, everything
appears to be working correctly so this is quite confusing.

Any help is appreciated.

Re: ResourceManager crash on deleted NM node back from the dead

2014-03-03 Thread Jian He 
Hi, I believe this is recently fixed in
https://issues.apache.org/jira/browse/YARN-713, and will be part of 2.4.0
release.

Jian


On Mon, Mar 3, 2014 at 5:19 AM, John Lilley wrote:

>  We had a DN/NM node that went offline for a while and been removed from
> the cluster via Ambari without decommissioning (because it was offline).
>
> When the node came back up, its NM attempted connection to the RM.
>
> Later the RM failed with this exception (dokken is the errant node):
>
>
>
> 2014-03-03 05:53:28,721 FATAL resourcemanager.ResourceManager
> (ResourceManager.java:run(449)) - Error in handling event type NODE_UPDATE
> to the scheduler
>
> java.lang.IllegalArgumentException: java.net.UnknownHostException:
> dokken.office.datalever.com
>
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
>
> at
> org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerToken(BuilderUtils.java:247)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager.createContainerToken(RMContainerTokenSecretManager.java:195)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainerToken(LeafQueue.java:1297)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1345)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1211)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1170)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:871)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:645)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:559)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:690)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:734)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:440)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.net.UnknownHostException: dokken.office.datalever.com
>
> ... 15 more
>
> 2014-03-03 05:53:28,723 INFO  resourcemanager.ResourceManager
> (ResourceManager.java:run(453)) - Exiting, bbye..
>
>
>
> I can see our error in how this node was removed while not being truly
> dead, but I think it shouldn't crash the RM.
>
> Forgive my ignorance ... do you know if this is a known problem?
>
>
>
> FYI:
>
> # hadoop version
>
> Hadoop 2.2.0.2.0.6.0-76
>
> Subversion g...@github.com:hortonworks/hadoop.git -r
> 8656b1cfad13b03b29e98cad042626205e7a1c86
>
> Compiled by jenkins on 2013-10-18T00:19Z
>
> Compiled with protoc 2.5.0
>
> From source with checksum d23ee1d271c6ac5bd27de664146be2
>
> This command was run using
> /usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-76.jar
>
>
>
> *Thanks*
>
> *John*
>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Not information in Job History UI

2014-03-03 Thread Jian He 
Note that node manager will not keep the finished applications and only
show running apps,  so the UI won't show the finished apps.
 Conversely, job history server UI will only show the finished apps but not
the running apps.

bq. there are no job logs generated on the server that is running the job.
by default, the local logs will be deleted after job finished.  you can
config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
the logs.

Jian


On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop  wrote:

> Hadoop 2.2.0
> CentOS 6.4
> Viewing UI in various browsers.
>
> I am having a problem where no information is visible in my Job History
> UI.  I run test jobs, they complete without error, but no information ever
> populates the nodemanager or jobhistory server UI.
>
> Also, there are no job logs generated on the server that is running the
> job.
>
> I have the following settings configured:
> yarn.nodemanager.local-dirs
> yarn.nodemanager.log-dirs
> yarn.log.server.url
>
> ...plus the basic yarn log dir.  I get output in regards to the daemons
> but very little in regards to the job.  All I get that refers to the
> jobhistory server is the following (so it appears to be functioning
> properly):
>
> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 19888
> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
> to /tmp/Jetty_server_19888_jobhistoryv7gnnv/webapp
> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
> SelectChannelConnector@server:19888
> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
> app /jobhistory started at 19888
> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
>
> I have a feeling this is a misconfiguration but I cannot figure out what
> setting is missing or wrong.
>
> Other than not being able to see any of the jobs in the UIs, everything
> appears to be working correctly so this is quite confusing.
>
> Any help is appreciated.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Not information in Job History UI

2014-03-03 Thread SF Hadoop 
Thanks for that info Jian.

You said, "there are no job logs generated on the server that is running
the job.".  So am I correct in assuming the logs will be in the dir
specified by yarn.nodemanager.log-dirs on the datanodes?

I am quite confused as to where the logs for each specific part of the
ecosystem reside.

If I were to run a job and I wanted to tail the job log as it was running,
where would I find that log?

Thanks for your help.


On Mon, Mar 3, 2014 at 11:46 AM, Jian He  wrote:

> Note that node manager will not keep the finished applications and only
> show running apps,  so the UI won't show the finished apps.
>  Conversely, job history server UI will only show the finished apps but
> not the running apps.
>
> bq. there are no job logs generated on the server that is running the job.
> by default, the local logs will be deleted after job finished.  you can
> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
> the logs.
>
> Jian
>
>
> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop  wrote:
>
>> Hadoop 2.2.0
>> CentOS 6.4
>> Viewing UI in various browsers.
>>
>> I am having a problem where no information is visible in my Job History
>> UI.  I run test jobs, they complete without error, but no information ever
>> populates the nodemanager or jobhistory server UI.
>>
>> Also, there are no job logs generated on the server that is running the
>> job.
>>
>> I have the following settings configured:
>> yarn.nodemanager.local-dirs
>> yarn.nodemanager.log-dirs
>> yarn.log.server.url
>>
>> ...plus the basic yarn log dir.  I get output in regards to the daemons
>> but very little in regards to the job.  All I get that refers to the
>> jobhistory server is the following (so it appears to be functioning
>> properly):
>>
>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 19888
>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>> to /tmp/Jetty_server_19888_jobhistoryv7gnnv/webapp
>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>> SelectChannelConnector@server:19888
>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>> app /jobhistory started at 19888
>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>> Registered webapp guice modules
>>
>> I have a feeling this is a misconfiguration but I cannot figure out what
>> setting is missing or wrong.
>>
>> Other than not being able to see any of the jobs in the UIs, everything
>> appears to be working correctly so this is quite confusing.
>>
>> Any help is appreciated.
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Need help to understand hadoop.tmp.dir

2014-03-03 Thread Chengwei Yang 
On Mon, Mar 03, 2014 at 09:03:28AM -0500, JCAD Cell 1 wrote:
> With the services stopped you would change the setting in core-site.xml:
>   
>     hadoop.tmp.dir
>     /var/hadoop/tmp
>   
> 
> Then move your /tmp/hadoop folder over to the new location:
> mv /tmp/hadoop /var/hadoop/tmp

Thank you, since the machine did a reboot, so all files in
/tmp/hadoop-user, the previous default direcotry of hadoop.tmp.dir, were
gone.

So that's the problem, no files to left, if the namenode can't re-create
its name space, then I think the only way is to format the namenode
again?

--
Thanks,
Chengwei

> 
> 
> 
> On Mon, Mar 3, 2014 at 5:55 AM, Chengwei Yang 
> wrote:
> 
> On Mon, Mar 03, 2014 at 01:57:49PM +0530, shashwat shriparv wrote:
> > NO need to format just change the value and restart the cluster;
> 
> Hmm, seems it doesn't work for me, if the only need to do is to change
> to another directory, then why it can not re-init the directory in /tmp
> just as another directory?
> 
> If I changed to another directory, a new directory, the same error
> happen.
> 
> --
> Thanks,
> Chengwei
> 
> >
> >
> > Warm Regards_∞_
> > Shashwat Shriparv
> > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> twitter.com/
> > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> +ShashwatShriparv
> > http://www.youtube.com/user/sShriparv/videoshttp://profile.yahoo.com/
> > SWXSTW3DVSDTF2HHSRM47AV6DI/
> >
> >
> >
> > On Mon, Mar 3, 2014 at 1:55 PM, Chengwei Yang 
>  >
> > wrote:
> >
> >     On Mon, Mar 03, 2014 at 11:56:08AM +0530, shashwat shriparv wrote:
> >     > Ya its always better to change the temp dir path in hadoop, as it
> will
> >     prevent
> >     > deletion of file while the server reboots.
> >
> >     Thanks, so is there anyway to recovery from this state? Or I have to
> format
> >     namenode again?
> >
> >     --
> >     Thanks,
> >     Chengwei
> >
> >     >
> >     >
> >     > Warm Regards_∞_
> >     > Shashwat Shriparv
> >     > http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9https://
> >     twitter.com/
> >     > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> >     +ShashwatShriparv
> >     > http://www.youtube.com/user/sShriparv/videoshttp://
> profile.yahoo.com/
> >     > SWXSTW3DVSDTF2HHSRM47AV6DI/
> >     >
> >     >
> >     >
> >     > On Mon, Mar 3, 2014 at 11:52 AM, Chengwei Yang <
> >     chengwei.yang...@gmail.com>
> >     > wrote:
> >     >
> >     >     On Mon, Mar 03, 2014 at 11:25:59AM +0530, shashwat shriparv
> wrote:
> >     >     > You can use any directory you like beside permissions are
> right.
> >     >
> >     >     I mean if it's better if we change the default hadoop.tmp.dir?
> >     Because it
> >     >     can not work cross reboot in default Linux environment.
> >     >
> >     >     --
> >     >     Thanks,
> >     >     Chengwei
> >     >
> >     >     >
> >     >     >
> >     >     > Warm Regards_∞_
> >     >     > Shashwat Shriparv
> >     >     > http://www.linkedin.com/pub/shashwat-shriparv/19/214/
> 2a9https://
> >     >     twitter.com/
> >     >     > shriparvhttps://www.facebook.com/shriparvhttp://google.com/
> >     >     +ShashwatShriparv
> >     >     > http://www.youtube.com/user/sShriparv/videoshttp://
> >     profile.yahoo.com/
> >     >     > SWXSTW3DVSDTF2HHSRM47AV6DI/
> >     >     >
> >     >     >
> >     >     >
> >     >     > On Mon, Mar 3, 2014 at 11:07 AM, Chengwei Yang <
> >     >     chengwei.yang...@gmail.com>
> >     >     > wrote:
> >     >     >
> >     >     >     Hi List,
> >     >     >
> >     >     >     I'm confusing by hadoop.tmp.dir currently because its
> default
> >     value
> >     >     >     "/tmp/hadoop-${user.name}" always means a directory in
> tmpfs in
> >     >     Linux.
> >     >     >     So after the name node machine reboot, it gone away and
> then
> >     name
> >     >     node
> >     >     >     fail to start.
> >     >     >
> >     >     >     I found this was reported here.
> >     >     >     
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user
> /
> >     201205.mbox
> >     >     /
> >     >     >     %3cbay148-w22bf95c5fbe2c40bf7cd9f86...@phx.gbl%3E
> >     >     >
> >     >     >     As I found from http://hadoop.apache.org/docs/r2.3.0/,
> there
> >     are a
> >     >     lot
> >     >     >     properties are based on hadoop.tmp.dir, like
> >     >     >     dfs.namenode.name.dir   
> file://${hadoop.tmp.dir}/dfs/name
> >     >     >
> >     >     >     I'm wondering, if we can set the def

Re: [hadoop] AvroMultipleOutputs org.apache.avro.file.DataFileWriter$AppendWriteException

2014-03-03 Thread Stanley Shi 
which avro version are you using when running outside of hadoop?

Regards,
*Stanley Shi,*



On Mon, Mar 3, 2014 at 11:49 PM, John Pauley wrote:

>   This is cross posted to avro-user list (
> http://mail-archives.apache.org/mod_mbox/avro-user/201402.mbox/%3ccf3612f6.94d2%25john.pau...@threattrack.com%3e
> ).
>
>   Hello all,
>
>  I’m having an issue using AvroMultipleOutputs in a map/reduce job.  The
> issue occurs when using a schema that has a union of null and a fixed
> (among other complex types), default to null, and it is not null.  Please
> find the full stack trace below and a sample map/reduce job that generates
> an Avro container file and uses that for the m/r input.  Note that I can
> serialize/deserialize without issue using
> GenericDatumWriter/GenericDatumReader outside of hadoop…  Any insight would
> be helpful.
>
>  Stack trace:
>  java.lang.Exception:
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: in com.foo.bar.simple_schema in union null
> of union in field baz of com.foo.bar.simple_schema
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
> Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: in com.foo.bar.simple_schema in union null
> of union in field baz of com.foo.bar.simple_schema
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296)
> at
> org.apache.avro.mapreduce.AvroKeyRecordWriter.write(AvroKeyRecordWriter.java:77)
> at
> org.apache.avro.mapreduce.AvroKeyRecordWriter.write(AvroKeyRecordWriter.java:39)
> at
> org.apache.avro.mapreduce.AvroMultipleOutputs.write(AvroMultipleOutputs.java:400)
> at
> org.apache.avro.mapreduce.AvroMultipleOutputs.write(AvroMultipleOutputs.java:378)
> at
> com.tts.ox.mapreduce.example.avro.AvroContainerFileDriver$SampleMapper.map(AvroContainerFileDriver.java:78)
> at
> com.tts.ox.mapreduce.example.avro.AvroContainerFileDriver$SampleMapper.map(AvroContainerFileDriver.java:62)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> Caused by: java.lang.NullPointerException: in com.foo.bar.simple_schema in
> union null of union in field baz of com.foo.bar.simple_schema
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:457)
> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:189)
> at org.apache.avro.reflect.ReflectData.isRecord(ReflectData.java:167)
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:608)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:265)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:597)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
> at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:175)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
>
>  Sample m/r job:
> 
>  package com.tts.ox.mapreduce.example.avro;
>
>  import org.apache.avro.Schema;
> import org.apache.avro.file.DataFileWriter;
> import org.apache.avro.generic.GenericData;
> import org.apache.avro.generic.GenericDatumWriter;
> import org.apache.avro.generic.GenericRecord;
> import org.apache.avro.generic.GenericRecordBuilder;
> import org.apache.avro.io.DatumWriter;
> import org.apache.avro.mapred.AvroKey;
> import org.apache.avro.mapreduce.AvroJob;
> import org.apache.avro.mapreduce.AvroKeyInputFormat;
> import org.apache.avro.mapreduce.AvroKeyOutputFormat;
> import org.apache.avro.mapreduce.AvroMultipleOutputs;
> import org.apache.hado

Re: class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final method getUnknownFields

2014-03-03 Thread Stanley Shi 
Why you have 2 hadoop version in the same pom file? In this case, you are
not going to know which hadoop class you are actually using.


  org.apache.hadoop
  hadoop-client
  2.3.0


org.apache.hadoop
hadoop-core
1.2.1




Regards,
*Stanley Shi,*



On Tue, Mar 4, 2014 at 1:15 AM, Margusja  wrote:

> Hi
>
> 2.2.0 and 2.3.0 gave me the same container log.
>
> A little bit more details.
> I'll try to use external java client who submits job.
> some lines from maven pom.xml file:
> 
>   org.apache.hadoop
>   hadoop-client
>   2.3.0
> 
> 
> org.apache.hadoop
> hadoop-core
> 1.2.1
> 
>
> lines from external client:
> ...
> 2014-03-03 17:36:01 INFO  FileInputFormat:287 - Total input paths to
> process : 1
> 2014-03-03 17:36:02 INFO  JobSubmitter:396 - number of splits:1
> 2014-03-03 17:36:03 INFO  JobSubmitter:479 - Submitting tokens for job:
> job_1393848686226_0018
> 2014-03-03 17:36:04 INFO  YarnClientImpl:166 - Submitted application
> application_1393848686226_0018
> 2014-03-03 17:36:04 INFO  Job:1289 - The url to track the job:
> http://vm38.dbweb.ee:8088/proxy/application_1393848686226_0018/
> 2014-03-03 17:36:04 INFO  Job:1334 - Running job: job_1393848686226_0018
> 2014-03-03 17:36:10 INFO  Job:1355 - Job job_1393848686226_0018 running in
> uber mode : false
> 2014-03-03 17:36:10 INFO  Job:1362 -  map 0% reduce 0%
> 2014-03-03 17:36:10 INFO  Job:1375 - Job job_1393848686226_0018 failed
> with state FAILED due to: Application application_1393848686226_0018 failed
> 2 times due to AM Container for appattempt_1393848686226_0018_02
> exited with  exitCode: 1 due to: Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
> Shell.java:589)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.
> launchContainer(DefaultContainerExecutor.java:195)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.
> launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.
> launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> ...
>
> Lines from namenode:
> ...
> 14/03/03 19:12:42 INFO namenode.FSEditLog: Number of transactions: 900
> Total time for transactions(ms): 69 Number of transactions batched in
> Syncs: 0 Number of syncs: 542 SyncTimes(ms): 9783
> 14/03/03 19:12:42 INFO BlockStateChange: BLOCK* addToInvalidates: blk_
> 1073742050_1226 90.190.106.33:50010
> 14/03/03 19:12:42 INFO hdfs.StateChange: BLOCK* allocateBlock:
> /user/hduser/input/data666.noheader.data. 
> BP-802201089-90.190.106.33-1393506052071
> blk_1073742056_1232{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]}
> 14/03/03 19:12:44 INFO hdfs.StateChange: BLOCK* InvalidateBlocks: ask
> 90.190.106.33:50010 to delete [blk_1073742050_1226]
> 14/03/03 19:12:53 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
> updated: 90.190.106.33:50010 is added to blk_1073742056
> _1232{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[
> ReplicaUnderConstruction[90.190.106.33:50010|RBW]]} size 0
> 14/03/03 19:12:53 INFO hdfs.StateChange: DIR* completeFile:
> /user/hduser/input/data666.noheader.data is closed by
> DFSClient_NONMAPREDUCE_-915999412_15
> 14/03/03 19:12:54 INFO BlockStateChange: BLOCK* addToInvalidates: blk_
> 1073742051_1227 90.190.106.33:50010
> 14/03/03 19:12:54 INFO hdfs.StateChange: BLOCK* allocateBlock:
> /user/hduser/input/data666.noheader.data.info. 
> BP-802201089-90.190.106.33-1393506052071
> blk_1073742057_1233{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]}
> 14/03/03 19:12:54 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
> updated: 90.190.106.33:50010 is added to blk_1073742057
> _1233{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[
> ReplicaUnderConstruction[90.190.106.33:50010|RBW]]} size 0
> 14/03/03 19:12:54 INFO hdfs.StateChange: DIR* completeFile:
> /user/hduser/input/data666.noheader.data.info is closed by
> DFSClient_NONMAPREDUCE_-915999412_15
> 14/03/03 19:12:55 INFO hdfs.StateChange: BLOCK* allocateBlock:
> /user/hduser/.staging/job_1393848686226_0019/job.jar.
> BP-802201089-90.190.106.33-1393506052071 blk_1073742058_1234{
> blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[
> ReplicaUnderConstruction[90.190.106.33:50010|RBW]]}

RE: class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final method getUnknownFields

2014-03-03 Thread Rohith Sharma K S 
Hi

  The reason for " 
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto overrides final 
method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet" is hadoop is 
compiled with protoc-2.5.0 version, but in the classpath lower version of 
protobuf is present.

1. Check MRAppMaster classpath, which version of protobuf is in classpath. 
Expected to have 2.5.0 version.
   

Thanks & Regards
Rohith Sharma K S



-Original Message-
From: Margusja [mailto:mar...@roo.ee] 
Sent: 03 March 2014 22:45
To: user@hadoop.apache.org
Subject: Re: class org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto 
overrides final method getUnknownFields

Hi

2.2.0 and 2.3.0 gave me the same container log.

A little bit more details.
I'll try to use external java client who submits job.
some lines from maven pom.xml file:
 
   org.apache.hadoop
   hadoop-client
   2.3.0
 
 
 org.apache.hadoop
 hadoop-core
 1.2.1
 

lines from external client:
...
2014-03-03 17:36:01 INFO  FileInputFormat:287 - Total input paths to process : 1
2014-03-03 17:36:02 INFO  JobSubmitter:396 - number of splits:1
2014-03-03 17:36:03 INFO  JobSubmitter:479 - Submitting tokens for job: 
job_1393848686226_0018
2014-03-03 17:36:04 INFO  YarnClientImpl:166 - Submitted application
application_1393848686226_0018
2014-03-03 17:36:04 INFO  Job:1289 - The url to track the job: 
http://vm38.dbweb.ee:8088/proxy/application_1393848686226_0018/
2014-03-03 17:36:04 INFO  Job:1334 - Running job: job_1393848686226_0018
2014-03-03 17:36:10 INFO  Job:1355 - Job job_1393848686226_0018 running in uber 
mode : false
2014-03-03 17:36:10 INFO  Job:1362 -  map 0% reduce 0%
2014-03-03 17:36:10 INFO  Job:1375 - Job job_1393848686226_0018 failed with 
state FAILED due to: Application application_1393848686226_0018 failed 2 times 
due to AM Container for
appattempt_1393848686226_0018_02 exited with  exitCode: 1 due to: 
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
 at org.apache.hadoop.util.Shell.run(Shell.java:379)
 at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
 at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
 at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
 at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)
...

Lines from namenode:
...
14/03/03 19:12:42 INFO namenode.FSEditLog: Number of transactions: 900 Total 
time for transactions(ms): 69 Number of transactions batched in
Syncs: 0 Number of syncs: 542 SyncTimes(ms): 9783
14/03/03 19:12:42 INFO BlockStateChange: BLOCK* addToInvalidates: 
blk_1073742050_1226 90.190.106.33:50010
14/03/03 19:12:42 INFO hdfs.StateChange: BLOCK* allocateBlock: 
/user/hduser/input/data666.noheader.data. 
BP-802201089-90.190.106.33-1393506052071
blk_1073742056_1232{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]}
14/03/03 19:12:44 INFO hdfs.StateChange: BLOCK* InvalidateBlocks: ask
90.190.106.33:50010 to delete [blk_1073742050_1226]
14/03/03 19:12:53 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated: 90.190.106.33:50010 is added to 
blk_1073742056_1232{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]} size 0
14/03/03 19:12:53 INFO hdfs.StateChange: DIR* completeFile: 
/user/hduser/input/data666.noheader.data is closed by
DFSClient_NONMAPREDUCE_-915999412_15
14/03/03 19:12:54 INFO BlockStateChange: BLOCK* addToInvalidates: 
blk_1073742051_1227 90.190.106.33:50010
14/03/03 19:12:54 INFO hdfs.StateChange: BLOCK* allocateBlock: 
/user/hduser/input/data666.noheader.data.info. 
BP-802201089-90.190.106.33-1393506052071
blk_1073742057_1233{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]}
14/03/03 19:12:54 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated: 90.190.106.33:50010 is added to 
blk_1073742057_1233{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[90.190.106.33:50010|RBW]]} size 0
14/03/03 19:12:54 INFO hdfs.StateChange: DIR* completeFile: 
/user/hduser/input/data666.noheader.data.info is closed by
DFSClient_NONMAPREDUCE_-915999412_15
14/03/03 19:12:55 INFO hdfs.StateChange: BLOCK* allocateBlock: 
/user/hduser/.staging/job_1393848686226_0019/job.jar. 
BP-802201089-90.190.1

Re: Not information in Job History UI

2014-03-03 Thread Jian He 
> You said, "there are no job logs generated on the server that is running
the job.".
that was quoting your previous sentence and answer your question..

> If I were to run a job and I wanted to tail the job log as it was
running, where would I find that log?
1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and
look for logs in local dirs specified by yarn.nodemanager.log-dirs.
Or
2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is
to aggregate those NM local logs and upload them to HDFS once application
is finished.Then you can use yarn logs command  or simply go the history UI
to see the logs.
You can find good explanation from
http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/

Thanks.


On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop  wrote:

> Thanks for that info Jian.
>
> You said, "there are no job logs generated on the server that is running
> the job.".  So am I correct in assuming the logs will be in the dir
> specified by yarn.nodemanager.log-dirs on the datanodes?
>
> I am quite confused as to where the logs for each specific part of the
> ecosystem reside.
>
> If I were to run a job and I wanted to tail the job log as it was running,
> where would I find that log?
>
> Thanks for your help.
>
>
> On Mon, Mar 3, 2014 at 11:46 AM, Jian He  wrote:
>
>> Note that node manager will not keep the finished applications and only
>> show running apps,  so the UI won't show the finished apps.
>>  Conversely, job history server UI will only show the finished apps but
>> not the running apps.
>>
>> bq. there are no job logs generated on the server that is running the
>> job.
>> by default, the local logs will be deleted after job finished.  you can
>> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
>> the logs.
>>
>> Jian
>>
>>
>> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop  wrote:
>>
>>> Hadoop 2.2.0
>>> CentOS 6.4
>>> Viewing UI in various browsers.
>>>
>>> I am having a problem where no information is visible in my Job History
>>> UI.  I run test jobs, they complete without error, but no information ever
>>> populates the nodemanager or jobhistory server UI.
>>>
>>> Also, there are no job logs generated on the server that is running the
>>> job.
>>>
>>> I have the following settings configured:
>>> yarn.nodemanager.local-dirs
>>> yarn.nodemanager.log-dirs
>>> yarn.log.server.url
>>>
>>> ...plus the basic yarn log dir.  I get output in regards to the daemons
>>> but very little in regards to the job.  All I get that refers to the
>>> jobhistory server is the following (so it appears to be functioning
>>> properly):
>>>
>>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>>> bound to port 19888
>>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>>> to /tmp/Jetty_server_19888_jobhistoryv7gnnv/webapp
>>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>>> SelectChannelConnector@server:19888
>>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>> app /jobhistory started at 19888
>>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>> Registered webapp guice modules
>>>
>>> I have a feeling this is a misconfiguration but I cannot figure out what
>>> setting is missing or wrong.
>>>
>>> Other than not being able to see any of the jobs in the UIs, everything
>>> appears to be working correctly so this is quite confusing.
>>>
>>> Any help is appreciated.
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Unable to export hadoop trunk into eclipse

2014-03-03 Thread nagarjuna kanamarlapudi 
Hi,
I checked out the hadoop trunck from
http://svn.apache.org/repos/asf/hadoop/common/trunk.

I set up protobuf-2.5.0 and then did mvn  build.
mvn clean install -DskipTests .. worked well. Maven build was Successful.

So, I tried importing the project into eclipse.

It is showing errors in pom.xml of hadoop-common project. Below are the
errors .. Can some one help me here ?

Plugin execution not covered by lifecycle configuration:
org.apache.hadoop:hadoop-maven-plugins:
 3.0.0-SNAPSHOT:version-info (execution: version-info, phase:
generate-resources


The error is at line 299  of pom.xml in hadoop-common project.


 
version-info
generate-resources

  version-info


  
${basedir}/src/main

  java/**/*.java
  proto/**/*.proto

  

  
  

There are multiple projects which failed of that error, hadoop-common is
one such project.

Regards,
Nagarjuna K

Re: Unable to export hadoop trunk into eclipse

2014-03-03 Thread Ted Yu 
Have you run the following command under the root of your workspace ?

mvn eclipse:eclipse 

On Mar 3, 2014, at 9:18 PM, nagarjuna kanamarlapudi 
 wrote:

> Hi,
> I checked out the hadoop trunck from 
> http://svn.apache.org/repos/asf/hadoop/common/trunk. 
> 
> I set up protobuf-2.5.0 and then did mvn  build. 
> mvn clean install -DskipTests .. worked well. Maven build was Successful. 
> 
> So, I tried importing the project into eclipse. 
> 
> It is showing errors in pom.xml of hadoop-common project. Below are the 
> errors .. Can some one help me here ? 
> 
> Plugin execution not covered by lifecycle configuration: 
> org.apache.hadoop:hadoop-maven-plugins:
>  3.0.0-SNAPSHOT:version-info (execution: version-info, phase: 
> generate-resources
> 
> 
> The error is at line 299  of pom.xml in hadoop-common project. 
> 
> 
>  
> version-info
> generate-resources
> 
>   version-info
> 
> 
>   
> ${basedir}/src/main
> 
>   java/**/*.java
>   proto/**/*.proto
> 
>   
> 
>   
>   
> 
> There are multiple projects which failed of that error, hadoop-common is one 
> such project.
> 
> Regards,
> Nagarjuna K
>

Re: Unable to export hadoop trunk into eclipse

2014-03-03 Thread nagarjuna kanamarlapudi 
Hi Ted,

I didn't do that earlier.

Now , I did it
mvn:eclipse eclipse
 and tried importing the projects same into eclipse. Now, this is throwing
the following errors


1. No marketplace entries found to handle Execution compile-protoc, in
hadoop-common/pom.xml in Eclipse.  Please see Help for more information.
2. No marketplace entries found to handle Execution compile-protoc, in
hadoop-hdfs/src/contrib/bkjournal/pom.xml in Eclipse.  Please see Help for
more information.


Any idea  ??


On Tue, Mar 4, 2014 at 10:59 AM, Ted Yu  wrote:

> Have you run the following command under the root of your workspace ?
>
> mvn eclipse:eclipse
>
> On Mar 3, 2014, at 9:18 PM, nagarjuna kanamarlapudi <
> nagarjuna.kanamarlap...@gmail.com> wrote:
>
> Hi,
> I checked out the hadoop trunck from
> http://svn.apache.org/repos/asf/hadoop/common/trunk.
>
> I set up protobuf-2.5.0 and then did mvn  build.
> mvn clean install -DskipTests .. worked well. Maven build was Successful.
>
> So, I tried importing the project into eclipse.
>
> It is showing errors in pom.xml of hadoop-common project. Below are the
> errors .. Can some one help me here ?
>
> Plugin execution not covered by lifecycle configuration:
> org.apache.hadoop:hadoop-maven-plugins:
>  3.0.0-SNAPSHOT:version-info (execution: version-info, phase:
> generate-resources
>
>
> The error is at line 299  of pom.xml in hadoop-common project.
>
>
>  
> version-info
> generate-resources
> 
>   version-info
> 
> 
>   
> ${basedir}/src/main
> 
>   java/**/*.java
>   proto/**/*.proto
> 
>   
> 
>   
>   
>
> There are multiple projects which failed of that error, hadoop-common is
> one such project.
>
> Regards,
> Nagarjuna K
>
>

Re: Unable to export hadoop trunk into eclipse

2014-03-03 Thread Azuryy Yu 
Have you installed protobuf on your computer?

https://code.google.com/p/protobuf/downloads/list



On Tue, Mar 4, 2014 at 3:08 PM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlap...@gmail.com> wrote:

> Hi Ted,
>
> I didn't do that earlier.
>
> Now , I did it
> mvn:eclipse eclipse
>  and tried importing the projects same into eclipse. Now, this is throwing
> the following errors
>
>
> 1. No marketplace entries found to handle Execution compile-protoc, in
> hadoop-common/pom.xml in Eclipse.  Please see Help for more information.
> 2. No marketplace entries found to handle Execution compile-protoc, in
> hadoop-hdfs/src/contrib/bkjournal/pom.xml in Eclipse.  Please see Help for
> more information.
>
>
> Any idea  ??
>
>
> On Tue, Mar 4, 2014 at 10:59 AM, Ted Yu  wrote:
>
>> Have you run the following command under the root of your workspace ?
>>
>> mvn eclipse:eclipse
>>
>> On Mar 3, 2014, at 9:18 PM, nagarjuna kanamarlapudi <
>> nagarjuna.kanamarlap...@gmail.com> wrote:
>>
>> Hi,
>> I checked out the hadoop trunck from
>> http://svn.apache.org/repos/asf/hadoop/common/trunk.
>>
>> I set up protobuf-2.5.0 and then did mvn  build.
>> mvn clean install -DskipTests .. worked well. Maven build was Successful.
>>
>> So, I tried importing the project into eclipse.
>>
>> It is showing errors in pom.xml of hadoop-common project. Below are the
>> errors .. Can some one help me here ?
>>
>> Plugin execution not covered by lifecycle configuration:
>> org.apache.hadoop:hadoop-maven-plugins:
>>  3.0.0-SNAPSHOT:version-info (execution: version-info, phase:
>> generate-resources
>>
>>
>> The error is at line 299  of pom.xml in hadoop-common project.
>>
>>
>>  
>> version-info
>> generate-resources
>> 
>>   version-info
>> 
>> 
>>   
>> ${basedir}/src/main
>> 
>>   java/**/*.java
>>   proto/**/*.proto
>> 
>>   
>> 
>>   
>>   
>>
>> There are multiple projects which failed of that error, hadoop-common is
>> one such project.
>>
>> Regards,
>> Nagarjuna K
>>
>>
>