I dived into the code and did a time breakdown for two read API.
The poor performance read invokes a method
fetchBlockByteRange(LocatedBlock block, long start, long end, byte[] buf,
int offset)
For the total 100 seconds read time, I measured the time running this
method, about 99686 ms.
In thi
I have a question that I think I already know the answer to but I would
like verification. I have a demo cluster comprised of two master nodes
and eight slaves (all 1x1.2 Ghz cpu / 1 Gig Ram / 1x250 Gig Sata 7200
rpm hard drives). I'm running small MR processes, about 100-200Gigs of
total dat
I have carefully examined the issue of versions to ensure that the hadoop
are tar from the same package, on the issue of block size, The system is
because I do a test, when I set the block size to 64MB, the front-end apache
can only read the document to reach 300-400kb, and I see my network traff
Hello Roman ,
If you have huge cluster then its good to have JobTracker and NameNode running
on different machines .
If your cluster is small enough ( ~<20-30 machines ) then you can run
JobTracker and NameNode on same machines .
Again it depends on hardware configuration . Usually NameNode and
Hello everyone,
is there any performance difference (or any advantage / disadvantage) in
colocating NameNode and JobTracker on the same node? Is it better to put
them on different nodes or on the same one?
Thank you for your answers.
Yours Sincerely,
Roman
What happens if you recompile with -verbose? Can you paste the output here?
- Aaron
On Sat, Jul 18, 2009 at 9:34 PM, Richard Bowser wrote:
> Sorry if I confused anyone, folks.
>
> My problem *description* differed from the example. My *source* code was
> identical.
>
> Here's my exact source cod
A VersionMismatch occurs because you're using different builds of Hadoop on
your different nodes. All DataNodes and the NameNode must be running the
exact same compilation of Hadoop (It's very strict).
One thing I noticed in your config is that you set dfs.block.size to
256. That's certainly n
There's likely another gotcha regarding the fact that various logs and job
config files are written to the _logs directory under the output directory.
You'd need to uniquify that as well. There may be other traps, but I don't
know them :)
This might be a bit of a frustrating endeavour since you're
;>> fsOwner=hadoopadmin,hadoopadmin
>>> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
>>> supergroup=supergroup
>>> 2009-07-20 14:57:41,869 INFO org.apache.hadoop.fs.FSNamesystem:
>>> isPermissionEnabled=true
>>> 2009-07-20 14:57:41,877
Hi Todd,
Thanks for the suggestions. I checked netstat -a on the master, and it
doesn't seem to indicate that port 50002 is in use by anybody:
r...@domu-12-31-39-04-30-16 (/vol/hadoop-0.20.0/)> netstat -a |more
Active Internet connections (servers and established)
Hi Jeyendran,
Is it possible that you've configured the jobtracker's RPC address
(mapred.job.tracker) to be the same as its HTTP address? The "Address
already in use" error indicates that someone is already claiming port 50002.
That might be another daemon on the same machine, or it could be anoth
Sorry, a typo in the previous post:
What I found running were:
Slaves: DataNode and TaskTracker are running
Master: Only NameNode is running
So everything except JobTracker on master are running.
Thanks,
jp
-Original Message-
From: Jeyendran Balakrishnan [mailto:jbalakrish...@docomolabs
Hello,
I downloaded Hadoop 0.20.0 and used the src/contrib/ec2/bin scripts to
launch a Hadoop cluster on Amazon EC2. To do so, I modified the bundled
scripts above for my EC2 account, and then created my own Hadoop 0.20.0
AMI. The steps I followed for creating AMIs and launching EC2 Hadoop
cluster
hadoop.dfs.Storage: Number of
files = 1808
2009-07-20 14:57:42,153 INFO org.apache.hadoop.dfs.Storage: Number of
files under construction = 1
2009-07-20 14:57:42,157 INFO org.apache.hadoop.dfs.Storage: Image file
of size 256399 loaded in 0 seconds.
2009-07-20 14:57:42,167 ERROR
org.apache.hadoop
ermissionEnabled=true
>> 2009-07-20 14:57:41,877 INFO org.apache.hadoop.dfs.FSNamesystemMetrics:
>> Initializing FSNamesystemMeterics using context
>> object:org.apache.hadoop.metrics.spi.NullContext
>> 2009-07-20 14:57:41,878 INFO org.apache.hadoop.fs.FSNamesystem:
>>
On 7/13/09 11:50 AM, "Dmitry Pushkarev" wrote:
> What I want is that 2TB of drive space per node were equally accessible to
> both tasktracker and datanode, and I'm not sure that mounting two drives to
> separate folders achieves that. (for example if reducer fills one drive
> will it start writi
systemStatusMBean
> 2009-07-20 14:57:41,908 INFO org.apache.hadoop.dfs.Storage: Number of
> files = 1808
> 2009-07-20 14:57:42,153 INFO org.apache.hadoop.dfs.Storage: Number of
> files under construction = 1
> 2009-07-20 14:57:42,157 INFO org.apache.hadoop.dfs.Storage: Image file
>
20 14:57:42,157 INFO org.apache.hadoop.dfs.Storage: Image file
of size 256399 loaded in 0 seconds.
2009-07-20 14:57:42,167 ERROR
org.apache.hadoop.dfs.LeaseManager:
/opt/hadoop/data/disk1/cs/raw/20090720/cs_2_20090720_17_1248113605290.data
not found in lease.paths
(=[/opt/hadoop/data/d
Hi Rakhi,
Currently Hadoop mapred/hdfs/common do not use Zookeeper. There are some
plans to use it in the JobTracker and NameNode but they are still being
discussed in the community. There are some jira's on hadoop that talk about
it.
http://issues.apache.org/jira/browse/MAPREDUCE-737
http://is
Hi,
I'm trying to run a few parallel jobs which have the same input directory
and the same output directory.
I modified the FileInputClass to check for non zero files, and also modified
the output class to allow non empty directories (the input directory =
output directory in my case). I made su
I don't think Zookeeper depends on hadoop, or vice-versa.
On 7/20/09 5:20 AM, "Rakhi Khatwani" wrote:
Hi,
I was going through Zookeeper and really interested in implementing
it, i am using hadoop-0.19.0 but coudnt find enough documentation which can
help me use zookeeper with hadoop-0.19
Martin,
You are comparing different things than in HDFS-236.
The '6x' difference you noted is for pread() with a random offset and
sequential offset (otherwise 6x difference is too small for sequential
and random access in general).
But what you are doing is comparing different read APIs, bot
>From your description, I'm not entirely clear on what tests you're
comparing. It seems almost like you're comparing your data on your hardware
against some data someone else posted on an entirely different cluster. Can
you upload a complete working Java program that reproduces all of your
results
ps. My Ubuntu version is* 8.04*
2009/7/20 George Pang
> Dear users,
>
> One question on the title on my hadoop web interface
> http://master:50060/tasktracker.jsp
> tracker_wpang-laptop:ip6-localhost/127.0.0.1:37496 Task Tracker Status
> After I disabled my ipv6 in /etc/modprobe.d/blacklist by a
Dear users,
One question on the title on my hadoop web interface
http://master:50060/tasktracker.jsp
tracker_wpang-laptop:ip6-localhost/127.0.0.1:37496 Task Tracker Status
After I disabled my ipv6 in /etc/modprobe.d/blacklist by adding these
lines:
# disable IPv6
blacklist ipv6
I don't know wh
You can do it from the API, I believe. Call getAssignedJobID() on
org.apache.hadoop.mapred.jobcontrol.Job to get the JobID of the job
you want to kill. Then call new JobClient().getJob(jobId).killJob().
On the new API you can just call killJob() on ControlledJob.
Tom
On Mon, Jul 20, 2009 at 4:45
Hi Tom,
in that case, can i kill the job by givin some command from the
API?? or i ll have 2 do it frm the command line?
On Mon, Jul 20, 2009 at 8:55 PM, Tom White wrote:
> Hi Raakhi,
>
> You can't suspend MapReduce jobs in Hadoop, which is why the
> JobControl API doesn't support jo
Hi Raakhi,
You can't suspend MapReduce jobs in Hadoop, which is why the
JobControl API doesn't support job suspension, only the ability to
kill jobs.
Cheers,
Tom
On Mon, Jul 20, 2009 at 9:39 AM, Rakhi Khatwani wrote:
> Hi,
> I have a scenario in which i have a list of 5 jobs. and an event
>
Hi,
I was going through Zookeeper and really interested in implementing
it, i am using hadoop-0.19.0 but coudnt find enough documentation which can
help me use zookeeper with hadoop-0.19.0
has anyone tried it with hadoop-0.19.0 or hadoop-0.19.1?
regards,
Raakhi
Hi Ari,
Thanks.
In Chukwa, how about the performance of the MapReduce jobs for merge.
The 1-hour merge and 1-day merge mapreduce jobs would run simultaneously,
how about the performance?
Schubert
On Sat, Jul 11, 2009 at 7:46 AM, Ariel Rabkin wrote:
> Chukwa uses a mapreduce job for this, with a
Waiting for response...
Thanks in advance.
Martin Mituzas wrote:
>
> hi, all
> I see there are two read in DFSInputStream:
>
> int read(byte buf[], int off, int len)
> int read(long position, byte[] buffer, int offset, int length)
>
> And I use the following code test the read performance.
Hi,
I have a scenario in which i have a list of 5 jobs. and an event
handler for example when triggered, would suspend all the running jobs.
but when i use a job control object, jobControl and execute,
jobControl.suspend(), it seems tht only the jobControl gets suspended and
not the 5 jobs wh
hi, all
I see there are two read in DFSInputStream:
int read(byte buf[], int off, int len)
int read(long position, byte[] buffer, int offset, int length)
And I use the following code test the read performance.
Before test I generate some files in the directory DATA_DIR, then I run this
functio
You would get "org.apache.hadoop.util.DiskChecker$DiskErrorException:
Could not find any valid local directory" if there is no space left on
disk to accomodate the file.
Thanks
Amareshwari
akhil1988 wrote:
Hi All,
I want to know why do we generally use tmp directory(and not any other) for
s
34 matches
Mail list logo
