Thanks, Steve Loughran. I learned sth from you!
2008/9/13 Steve Loughran <[EMAIL PROTECTED]>
> 叶双明 wrote:
>
>> Thanks for paying attention to my tentative idea!
>>
>> What I thought isn't how to store the meradata, but the final (or last)
>> way
>> to recover valuable data in the cluster when so
叶双明 wrote:
Thanks for paying attention to my tentative idea!
What I thought isn't how to store the meradata, but the final (or last) way
to recover valuable data in the cluster when something worst (which destroy
the metadata in all multiple NameNode) happen. i.e. terrorist attack or
natural d
Thanks.
It seem that it isn't a right way, but i learn a lot from you.
2008/9/12 Pete Wyckoff <[EMAIL PROTECTED]>
>
> You may want to look at hadoop's proposal for snapshotting, where one can
> take a snapshot's metadata and store it in some disaster resilient place(s)
> for a rainy day:
>
> htt
You may want to look at hadoop's proposal for snapshotting, where one can
take a snapshot's metadata and store it in some disaster resilient place(s)
for a rainy day:
https://issues.apache.org/jira/browse/HADOOP-3637
On 9/11/08 10:06 AM, "Dhruba Borthakur" <[EMAIL PROTECTED]> wrote:
> My op
My opinion is to not store file-namespace related metadata on the
datanodes. When a file is renamed, one has to contact all datanodes to
change this new metadata. Worse still, if one renames an entire
subdirectory, all blocks that belongs to all files in the subdirectory
have to be updated. Similar
Thanks Ari Rabkin!
1. I think the cost is very low, if the block's size is 10m, 1k/10m almost
0.01% of the disk space.
2. Actually, if two of racks lose and replication <= 3, it seem that we
can't recover all data. But in the situation of losing one rack of two racks
and replication >=2, we can r
I don't understand this use case.
Suppose that you lose half the nodes in the cluster. On average,
12.5% of your blocks were exclusively stored on the half the cluster
that's dead. For many (most?) applications, a random 87.5% of the
data isn't really useful. Storing metadata in more places wou
ith metadata...
>
> -Original Message-
> From: 叶双明 [mailto:[EMAIL PROTECTED]
> Sent: Wednesday, September 10, 2008 12:06 AM
> To: core-user@hadoop.apache.org
> Subject: Re: Thinking about retriving DFS metadata from datanodes!!!
>
> I think let each block carry three simple additi
retriving DFS metadata from datanodes!!!
I think let each block carry three simple additional information which
doesn't use in normal situation:
1. which file that it belong to
2. which block is it in the file
3. how many blocks of the file
After the cluster system has been destroy, we ca
I think let each block carry three simple additional information which
doesn't use in normal situation:
1. which file that it belong to
2. which block is it in the file
3. how many blocks of the file
After the cluster system has been destroy, we can set up new NameNode , and
then , rebuild
The main problem is the complexity of maintaining accuracy of the
metadata. In other words, what you think is the cost?
Do you think writing fsimage to multiple places helps with the terrorist
attack? It is supported even now.
Raghu.
叶双明 wrote:
Thanks for paying attention to my tentative
Thanks for paying attention to my tentative idea!
What I thought isn't how to store the meradata, but the final (or last) way
to recover valuable data in the cluster when something worst (which destroy
the metadata in all multiple NameNode) happen. i.e. terrorist attack or
natural disasters dest
+1 -
from the perspective of the data nodes, dfs is just a block-level store and
is thus much more robust and scalable.
On 9/9/08 9:14 AM, "Owen O'Malley" <[EMAIL PROTECTED]> wrote:
> This isn't a very stable direction. You really don't want multiple distinct
> methods for storing the metadat
This isn't a very stable direction. You really don't want multiple distinct
methods for storing the metadata, because discrepancies are very bad. High
Availability (HA) is a very important medium term goal for HDFS, but it will
likely be done using multiple NameNodes and ZooKeeper.
-- Owen
I am thinking about that, let datanodes carry additional information about
which file that it belong to, sequence number in the order of blocks, and
how many blocks of the file.
Is it make sense?
2008/9/9 Raghu Angadi <[EMAIL PROTECTED]>
>
> There is certainly value in it. But it can not be done
There is certainly value in it. But it can not be done with the data
datanodes currently have.
The usual way to protect NameNode metadata in Hadoop is to write the
metadata to two different locations.
Raghu.
叶双明 wrote:
Hi all.
We all know the importance of NameNode for a cluster, when th
Hi all.
We all know the importance of NameNode for a cluster, when the NameNode
whith all matadata of the DFS is breaked down, the whole of the
clustes'data is gone.
So, if we can retrive DFS metadata from datanodes, It should be a additional
great robustness for hadoop.
Does any one thinking I
17 matches
Mail list logo
