Re: How many real cores in large type instances on EC2 ?

[Harsh J]

Jiamin,

Agree it is. I just wanted you to head to the right source for the proper 
answer :)

Here are the cpuinfos of each instance, so be assured you're not being given 
anything lower: 
http://www.cloudiquity.com/2009/01/amazon-ec2-instances-and-cpuinfo/

On 25-Nov-2011, at 12:01 PM, Jiamin Lu wrote:

> Dear Harsh, 
> 
> Thanks for your reply and suggestion. 
> I will read your recommendation carefully. 
> If I find something important, I will left my answer to this thread too, 
> since I think EC2 is quite important in parallel processing. 
> 
> Thanks
> 
> 
> On Thu, Nov 24, 2011 at 6:31 PM, Harsh J  wrote:
> Jiamin,
> 
> Does AMZN not carry a forum of their own for these questions? They'd
> be the best to ask really.
> 
> I do not know what an "EC2 Compute Unit" means, but the page clearly
> says two virtual cores. Perhaps this may help understand:
> http://gevaperry.typepad.com/main/2009/03/figuring-out-the-roi-of-infrastructureasaservice.html
> 
> On Thu, Nov 24, 2011 at 9:10 PM, Jiamin Lu  wrote:
> > Hi, all
> >
> > I am using the Amazon EC2, with their large instances.
> > Amazon claims these large type instances have 4 EC2 Compute units (2 virtual
> > cores with 2 EC2 Compute Units each).
> > But according to my observation, it seems like they only have two cores.
> >
> > I checked the /proc/cpuinfo, which shows there are only two processors,
> > I also used the top command, and it also says only two cpu there.
> >
> > Can someone tell me actually how many cores are contained inside these large
> > instances?
> > Did I misunderstand these terms that Amazon talks about ??
> >
> > Thanks
> >
> > Jiamin Lu
> >
> >
> 
> 
> 
> --
> Harsh J
> 
> 
> 
> -- 
> Jiamin Lu
> Fernuniversität Hagen
> D-58084 Hagen
> GERMANY
> 
> Phone: +49-2331-987-4276
> Email: jiamin@gmail.com

Re: How many real cores in large type instances on EC2 ?

[Robert Hafner]

You're looking at two different things. The number of cores is one thing, and 
the power of each core another. Number of cores is pretty straight forward, and 
the power is relative to that of a small instance. You've got the answer right 
there in your email- there are two virtual cores of two compute units each, so 
you will only see two cores. Those cores are twice as powerful as the typical 
core (hence the two compute units), but they are still only going to show two 
cores.

This does have a bit of an effect on processing jobs. If you need more CPU 
power behind your tasks than a large instance will be nice, but if your 
bottleneck is something else (memory, disk i/o, s3 access, etc) then you'll 
probably be optimizing for the wrong problem. In my experience it's better to 
have four small instances running in a job than one large instance, even though 
the cost is equivalent. If you're using spot instances- which are a huge budget 
saver- then the small instances make even more sense, since lost machines will 
only result in one lost task instead of four.

Robert


On Nov 24, 2011, at 7:40 AM , Jiamin Lu wrote:

> Hi, all
> 
> I am using the Amazon EC2, with their large instances. 
> Amazon claims these large type instances have 4 EC2 Compute units (2 virtual 
> cores with 2 EC2 Compute Units each). 
> But according to my observation, it seems like they only have two cores. 
> 
> I checked the /proc/cpuinfo, which shows there are only two processors, 
> I also used the top command, and it also says only two cpu there. 
> 
> Can someone tell me actually how many cores are contained inside these large 
> instances? 
> Did I misunderstand these terms that Amazon talks about ??
> 
> Thanks
> 
> Jiamin Lu
> 

Registration of Node Manger Failed

[sri ram]

Hi,
   I try to install hadoop 0.23 and form a small cluster with 3
machines.
   Whenever i try to start nodemanager and resource manager.The
nodemanager fails to start by throwing the following error log.And the
nodemanager fails in both master and slaves.

2011-11-25 13:40:15,244 INFO  service.AbstractService
(AbstractService.java:start(61)) - Service:Dispatcher is started.
2011-11-25 13:40:15,244 INFO  ipc.YarnRPC (YarnRPC.java:create(47)) -
Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
2011-11-25 13:40:15,246 INFO  ipc.HadoopYarnRPC
(HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc
proxy for protocol interface
org.apache.hadoop.yarn.server.api.ResourceTracker
2011-11-25 13:40:15,289 INFO  nodemanager.NodeStatusUpdaterImpl
(NodeStatusUpdaterImpl.java:registerWithRM(155)) - Connected to
ResourceManager at master:8025
2011-11-25 13:40:15,407 ERROR service.CompositeService
(CompositeService.java:start(72)) - Error starting services
org.apache.hadoop.yarn.server.nodemanager.NodeManager
org.apache.avro.AvroRuntimeException: org.apache.hadoop.yarn.YarnException:
Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager
failed
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:132)
at
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:231)
Caused by: org.apache.hadoop.yarn.YarnException: Recieved SHUTDOWN signal
from Resourcemanager ,Registration of NodeManager failed
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:165)
at
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:128)
... 3 more
2011-11-25 13:40:15,408 INFO  event.AsyncDispatcher
(AsyncDispatcher.java:run(71)) - AsyncDispatcher thread interrupted
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:69)
at java.lang.Thread.run(Thread.java:636)
2011-11-25 13:40:15,410 INFO  service.AbstractService
(AbstractService.java:stop(75)) - Service:Dispatcher is stopped.
2011-11-25 13:40:15,470 INFO  mortbay.log (Slf4jLog.java:info(67)) -
Stopped SelectChannelConnector@0.0.0.0:
2011-11-25 13:40:15,588 INFO  service.AbstractService
(AbstractService.java:stop(75)) -
Service:org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer is
stopped.
2011-11-25 13:40:15,589 INFO  ipc.Server (Server.java:stop(1709)) -
Stopping server on 59072
2011-11-25 13:40:15,589 INFO  ipc.Server (Server.java:run(1533)) - IPC
Server handler 0 on 59072: exiting
2011-11-25 13:40:15,590 INFO  ipc.Server (Server.java:run(1533)) - IPC
Server handler 1 on 59072: exiting
2011-11-25 13:40:15,590 INFO  ipc.Server (Server.java:run(1533)) - IPC
Server handler 2 on 59072: exiting
2011-11-25 13:40:15,591 INFO  ipc.Server (Server.java:run(1533)) - IPC
Server handler 3 on 59072: exiting
2011-11-25 13:40:15,591 INFO  ipc.Server (Server.java:run(1533)) - IPC
Server handler 4 on 59072: exiting
2011-11-25 13:40:15,593 INFO  ipc.Server (Server.java:run(495)) - Stopping
IPC Server listener on 59072
2011-11-25 13:40:15,594 INFO  service.AbstractService
(AbstractService.java:stop(75)) -
Service:org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
is stopped.
2011-11-25 13:40:15,600 INFO  event.AsyncDispatcher
(AsyncDispatcher.java:run(71)) - AsyncDispatcher thread interrupted
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:69)
at java.lang.Thread.run(Thread.java:636)
2011-11-25 13:40:15,601 INFO  ipc.Server (Server.java:run(637)) - Stopping
IPC Server Responder
2011-11-25 13:40:15,601 INFO  service.AbstractService
(AbstractService.java:stop(75)) - Service:Dispatcher is stopped.
2011-11-25 13:40:15,602 WARN  monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(464)) -
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
is interrupted. Exiting.
2011-11-25 13:40:15,602 INFO  ser

Creating a custom trace file using Rumen

[arun k]

Hi guys !

I was trying to use Rumen to generate trace files.  I have few queries :

Q1> Is there any way to create a new trace file from job history logs with
custom set of split locations ?

Q2> Can we create new trace files from existing trace files with new values
for the attributes like preferred locations ?

Q3> How can i add new attributes / fields (which are not in job history
logs) to the job or the tasks in the trace ? (or)

  Is there any way to generate trace with extra fields ?

Can anyone help me in this regard ?


Thanks,
Arun