FYI, Just ran a 50 node cluster using one of the new kernels for
Fedora with all nodes forced onto the same 'availability zone' and
there were no timeouts or failed writes.
On Mar 27, 2008, at 4:16 PM, Chris K Wensel wrote:
If it's any consolation, I'm seeing similar behaviors on 0.16.0 when
On Fri, Mar 28, 2008 at 12:31 AM, 朱盛凯 <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I met this problem in my cluster before, I think I can share with you some
> of my experience.
> But it may not work in you case.
>
> The job in my cluster always hung at 16% of reduce. It occured because the
> reduce task
never mind. i found archive.apache.org.
On Mar 27, 2008, at 6:00 PM, Chris K Wensel wrote:
is this still true?
if so, can we restore 0.16.0 at http://www.apache.org/dist/hadoop/core/
?
just realized it was missing as I am rebuilding my ec2 ami's.
On Mar 17, 2008, at 8:43 PM, Owen O'Malley
is this still true?
if so, can we restore 0.16.0 at http://www.apache.org/dist/hadoop/
core/ ?
just realized it was missing as I am rebuilding my ec2 ami's.
On Mar 17, 2008, at 8:43 PM, Owen O'Malley wrote:
We believe that there has been a regression in release 0.16.1 with
respect to the re
If it's any consolation, I'm seeing similar behaviors on 0.16.0 when
running on EC2 when I push the number of nodes in the cluster past 40.
On Mar 24, 2008, at 6:31 AM, André Martin wrote:
Thanks for the clarification, dhruba :-)
Anyway, what can cause those other exceptions such as "Could no
Shouldn't be any issues with this. It's similar to the setup we have
right now.
-Xavier
-Original Message-
From: Colin Freas
Sent: Thursday, March 27, 2008 12:05 PM
To: Hadoop
Subject: nfs mount hadoop-site?
are there any issues with having the hadoop-site.xml in .../conf placed
on an
We looked seriously at HDFS and MogileFS and considered (and instantly
rejected a *bunch* of others).
HDFS was eliminated based on number of files, lack of HA and lack of
reference implementations serving large scale web sites directly from it.
Mogile had HA (using crude tools), reference implem
Hello,
I'm working on large amount of logs, and I've noticed that the
distribution of data on the network (./hadoop dfs -put input input)
takes a lot of time.
Let's says that my data is already distributed among the network, is
there anyway to say to hadoop to use the already existing
distributio
are there any issues with having the hadoop-site.xml in .../conf placed on
an nfs mounted dir that all my nodes have access to?
-colin
Robert Krüger wrote:
this seems like an FAQ but I didn't explicitly see it in the docs: Is
the minmium size a file occupies on HDFS controlled by the block size,
i.e. would using the default block size of 64 B result in consumption of
64 MB if I stored a file of 1 byte?
No. The last block in
Hi,
this seems like an FAQ but I didn't explicitly see it in the docs: Is
the minmium size a file occupies on HDFS controlled by the block size,
i.e. would using the default block size of 64 B result in consumption of
64 MB if I stored a file of 1 byte? I would assume yes based on the fact
t
might be off-topic but how would you compare GlusterFS to HDFS and
MogileFS for such an application? Did you look at that at all and
decided against it?
Ted Dunning wrote:
We evaluated several options for just this problem and eventually settled on
MogileFS. That said, Mogile needed severa
We evaluated several options for just this problem and eventually settled on
MogileFS. That said, Mogile needed several weeks of work to get it ready
for prime time. It will work pretty well for modest sized collections, but
for our stuff (many hundreds of millions of files, approaching PB of
st
Yes.
The present work-arounds for this are pretty complicated.
option1) you can write small files relatively frequently and every time you
write some number of them, you can concatenate them and delete them. These
concatenations can receive the same treatment. If managed carefully in
conjunct
I can't offer any insights into other clustering FS solutions, but I
think it's a very safe bet to say that Google relies entirely on GFS
for their long-term storage. Granted, they almost certainly make
offline backups of business-critical data, but I would assume that
everything related to GMail,
Hi,
I met this problem in my cluster before, I think I can share with you some
of my experience.
But it may not work in you case.
The job in my cluster always hung at 16% of reduce. It occured because the
reduce task could not fetch the
map output from other nodes.
In my case, two factors may re
On Thu, 27 Mar 2008, Natarajan, Senthil wrote:
> Hi,
> I have small Hadoop cluster, one master and three slaves.
> When I try the example wordcount on one of our log file (size ~350 MB)
>
> Map runs fine but reduce always hangs (sometime around 19%,60% ...) after
> very long time it finishes.
> I
Hi,
I have small Hadoop cluster, one master and three slaves.
When I try the example wordcount on one of our log file (size ~350 MB)
Map runs fine but reduce always hangs (sometime around 19%,60% ...) after very
long time it finishes.
I am seeing this error
Shuffle Error: Exceeded MAX_FAILED_UNIQ
It is a good place to start
http://wiki.apache.org/hadoop/GettingStartedWithHadoop
Check the articles which descrive how to set up a sigle node and a two
cluster node.
http://wiki.apache.org/hadoop/HadoopArticles
I followed them and it work! ;)
The two cluster node installation also scales to mor
I am experiencing bind errors in my log files when trying to start up the
hadoop cluster.
However, the log files do not really show me which adress/port pair is
responsible for the error.
I tried some property settings in the hadoop-site.xml but actually I don't
know which setting is responsible
Hello,
where can I find a complete list of all possible configuration properties
(e.g. in file hadoop-site.xml)?
I am experiencing lots of bind errors in my log-files when trying to
start-dfs.sh, start-mapred.sh, start-all.sh!
That's why I think, I have to change some port settings. But I don't
Hi,
we're looking for options for creating a scalable storage solution based
on commodity hardware for media files (spacewise dominated video files
of a few hundred MB but also to store up to a few million smaller files
such as thumbnails). The system will start with a few TB and should be
a
Hi,
Thanks for the reply.
Does this mean that once I close a file, I can open it only for reading?
And if I reopen the same file to write any data then the old data will be
lost and again its as good as a new file being created with the same name?
On Thu, Mar 27, 2008 at 12:23 PM, dhruba Borthakur
23 matches
Mail list logo
