Question regarding hdfs synchronously / asynchronously block replication

[Ronen Itkin]

Hi all!

My question is regarding hdfs block replication.
>From the perspective of client, does the application receives an ACK for a
certain packet after it was written on the first
hadoop data node in the pipeline? or after the packet is *replicated* to all
assigned *replication* nodes?

More generaly, does Hadoop's HDFS block replication works synchronously or
asynchronously?

synchronously --> more replications =  decrease in write performances
(client has to wait until every packet will be written to all replication
nodes before he receives an ACK).
asynchronously --> more replication has no influence on write performance
(client recieves an ACK packet after the first write to the first datadone
finishes, hdfs will complete its replication on his free time).

synchronously / asynchronously block replication - is it something
configurable ? If it is, than how can I do it?

Thanks!

-- 
*
Ronen Itkin*
Taykey | www.taykey.com

Re: Question regarding hdfs synchronously / asynchronously block replication

[Ronen Itkin]

Thanks a lot!!

On Wed, Oct 5, 2011 at 3:51 PM, Eric Fiala  wrote:

> Ronen,
> On file write HDFS's block replication pipeline is asynchronous - datanode
> 1
> gets a block before passing it onto datanode 2, and so on (limiting network
> traffic between client node and the data nodes - it only writes to one).
>
> The ACK for a packet is returned only once all datanodes in the pipeline
> have copied the block.
>
> However, if a failure occurs in the interim on a datanode in the write
> pipeline, AND the minimum replication threshold has been met (normally 1) -
> namenode will, in seperate operation, quell the replica deficit.
>
> Don't think that's configurable, however, it would be interesting use case
> for speeding up writes, while trading off some reliability.
>
> EF
>
> On Wed, Oct 5, 2011 at 1:53 AM, Ronen Itkin  wrote:
>
> > Hi all!
> >
> > My question is regarding hdfs block replication.
> > From the perspective of client, does the application receives an ACK for
> a
> > certain packet after it was written on the first
> > hadoop data node in the pipeline? or after the packet is *replicated* to
> > all
> > assigned *replication* nodes?
> >
> > More generaly, does Hadoop's HDFS block replication works synchronously
> or
> > asynchronously?
> >
> > synchronously --> more replications =  decrease in write performances
> > (client has to wait until every packet will be written to all replication
> > nodes before he receives an ACK).
> > asynchronously --> more replication has no influence on write performance
> > (client recieves an ACK packet after the first write to the first
> datadone
> > finishes, hdfs will complete its replication on his free time).
> >
> > synchronously / asynchronously block replication - is it something
> > configurable ? If it is, than how can I do it?
> >
> > Thanks!
> >
> > --
> > *
> > Ronen Itkin*
> > Taykey | www.taykey.com
> >
>



-- 
*
Ronen Itkin*
Taykey | www.taykey.com

HDFS DataNode daily log growing really high and fast

[Ronen Itkin]

Hey all!

I am having an issue with hadoop's daily datanode log growing to + 1.8 GB.
I have 3 Nodes in my hdfs cluster, all sharing the same configuration
(including same log4j.properties).
While running operations and jobs equally (automatically) on whole of the
nodes, only one of them (data node*03*)is having this issue with the log
growing high.

/var/log/hadoop/hadoop-hadoop-datanode-ip-10-10-10-4.log

The log does not show any exceptions, just many hdfs operations
(read+write).

I am currently running *Cloudera* *hadoop-0.20.2-cdh3u1* and this is my
architecture:

*MasterServer*: NameNode, JobTracker, HBase HMaster (*hbase-0.90.3-cdh3u1*),
Zookeeper01
*Node01*: Data Node, TaskTracker, HBase HRegion
*Node02*: Data Node, TaskTracker, HBase HRegion
*Node03*: Data Node, TaskTracker, HBase HRegion
*SecondaryServer*: Secondary NameNode, HBase Backup HMaster, Zookeeper02
*ServerX*: ZooKeeper03


Does any one can think of a good reason, why it happens ? why to a specific
node? is is related to hbase operations? hdfs block scanner?

Here is a sample of the log file:

2011-10-30 08:52:27,313 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /*
10.10.10.4*:50010, dest: /*10.10.10.4*:43447, bytes: 66564, op: HDFS_READ,
cliID: 
DFSClient_hb_rs_ip-*10-10-10-4*.ec2.internal,60020,1318334166605_1318334167243,
offset: 34500096, srvID: DS-75443592-10.93.67.113-50010-1318335522512,
blockid: blk_2773771462926694276_25674, duration: 274702
2011-10-30 08:52:27,314 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /*
10.10.10.4*:50010, dest: /*10.10.10.4*:43448, bytes: 66564, op: HDFS_READ,
cliID: 
DFSClient_hb_rs_ip-*10-10-10-4*.ec2.internal,60020,1318334166605_1318334167243,
offset: 34631168, srvID: DS-75443592-10.93.67.113-50010-1318335522512,
blockid: blk_2773771462926694276_25674, duration: 236691



Thanks,
*Ronen.*

 

Re: HDFS DataNode daily log growing really high and fast

[Ronen Itkin]

For instance, yesterday's daily log:
"/var/log/hadoop/hadoop-hadoop-datanode-ip-10-10-10-4.log"
on the problematic Node03 was in the size of 1.1 GB while on other Nodes
the same log was in the size of 87 MB.

Again, nothing is being run specifically on Node03, I have 3 nodes, with
replication of 3 - means that all the data is being saved on every node,
All nodes are connected to the same switch (and on the same subnet) - so no
advantages to Node03 in any Job.

I am being suspicious regarding HBase...

What do you think? any suggestions?

Thank a head!




On Sun, Oct 30, 2011 at 11:02 AM, Ronen Itkin  wrote:

> Hey all!
>
> I am having an issue with hadoop's daily datanode log growing to + 1.8 GB.
> I have 3 Nodes in my hdfs cluster, all sharing the same configuration
> (including same log4j.properties).
> While running operations and jobs equally (automatically) on whole of the
> nodes, only one of them (data node*03*)is having this issue with the log
> growing high.
>
> /var/log/hadoop/hadoop-hadoop-datanode-ip-10-10-10-4.log
>
> The log does not show any exceptions, just many hdfs operations
> (read+write).
>
> I am currently running *Cloudera* *hadoop-0.20.2-cdh3u1* and this is my
> architecture:
>
> *MasterServer*: NameNode, JobTracker, HBase HMaster (*hbase-0.90.3-cdh3u1*),
> Zookeeper01
> *Node01*: Data Node, TaskTracker, HBase HRegion
> *Node02*: Data Node, TaskTracker, HBase HRegion
> *Node03*: Data Node, TaskTracker, HBase HRegion
> *SecondaryServer*: Secondary NameNode, HBase Backup HMaster, Zookeeper02
> *ServerX*: ZooKeeper03
>
>
> Does any one can think of a good reason, why it happens ? why to a
> specific node? is is related to hbase operations? hdfs block scanner?
>
> Here is a sample of the log file:
>
> 2011-10-30 08:52:27,313 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /*
> 10.10.10.4*:50010, dest: /*10.10.10.4*:43447, bytes: 66564, op:
> HDFS_READ, cliID: 
> DFSClient_hb_rs_ip-*10-10-10-4*.ec2.internal,60020,1318334166605_1318334167243,
> offset: 34500096, srvID: DS-75443592-10.93.67.113-50010-1318335522512,
> blockid: blk_2773771462926694276_25674, duration: 274702
> 2011-10-30 08:52:27,314 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /*
> 10.10.10.4*:50010, dest: /*10.10.10.4*:43448, bytes: 66564, op:
> HDFS_READ, cliID: 
> DFSClient_hb_rs_ip-*10-10-10-4*.ec2.internal,60020,1318334166605_1318334167243,
> offset: 34631168, srvID: DS-75443592-10.93.67.113-50010-1318335522512,
> blockid: blk_2773771462926694276_25674, duration: 236691
>
>
>
> Thanks,
> *Ronen.*
>
>  <http://www.taykey.com/>
>
>


-- 
*
Ronen Itkin*
Taykey | www.taykey.com

Re: HDFS DataNode daily log growing really high and fast

[Ronen Itkin]

Hey,

 It has more Regions, I think that there is a "hot spot / hot region" that
is being read a lot more that others.
How can I recognize this kind of region, in order to administratively split
them? are there any tools for that?

Thanks

On Mon, Oct 31, 2011 at 2:27 PM, Brock Noland  wrote:

> Hi,
>
> On Mon, Oct 31, 2011 at 12:59 AM, Ronen Itkin  wrote:
> > For instance, yesterday's daily log:
> > "/var/log/hadoop/hadoop-hadoop-datanode-ip-10-10-10-4.log"
> > on the problematic Node03 was in the size of 1.1 GB while on other Nodes
> > the same log was in the size of 87 MB.
> >
> > Again, nothing is being run specifically on Node03, I have 3 nodes, with
> > replication of 3 - means that all the data is being saved on every node,
> > All nodes are connected to the same switch (and on the same subnet) - so
> no
> > advantages to Node03 in any Job.
> >
> > I am being suspicious regarding HBase...
>
>
> Does that servers regionserver have more regions assigned to it?
> Check the HBase GUI.
>
> Also, you can turn that message off with:
>
>
> log4j.logger.org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace=WARN
>
>
> Brock
>



-- 
*
Ronen Itkin*
Taykey | www.taykey.com

Re: Timer jobs

[Ronen Itkin]

Hi

Try to use Oozie for job coordination and work flows.



On Thu, Sep 1, 2011 at 12:30 PM, Per Steffensen  wrote:

> Hi
>
> I use hadoop for a MapReduce job in my system. I would like to have the job
> run very 5th minute. Are there any "distributed" timer job stuff in hadoop?
> Of course I could setup a timer in an external timer framework (CRON or
> something like that) that invokes the MapReduce job. But CRON is only
> running on one particular machine, so if that machine goes down my job will
> not be triggered. Then I could setup the timer on all or many machines, but
> I would not like the job to be run in more than one instance every 5th
> minute, so then the timer jobs would need to coordinate who is actually
> starting the job "this time" and all the rest would just have to do nothing.
> Guess I could come up with a solution to that - e.g. writing some "lock"
> stuff using HDFS files or by using ZooKeeper. But I would really like if
> someone had already solved the problem, and provided some kind of a
> "distributed timer framework" running in a "cluster", so that I could just
> register a timer job with the cluster, and then be sure that it is invoked
> every 5th minute, no matter if one or two particular machines in the cluster
> is down.
>
> Any suggestions are very welcome.
>
> Regards, Per Steffensen
>



-- 
*
Ronen Itkin*
Taykey | www.taykey.com

Re: Timer jobs

[Ronen Itkin]

If I get you right you are asking about Installing Oozie as Distributed
and/or HA cluster?!
In that case I am not familiar with an out of the box solution by Oozie.
But, I think you can made up a solution of your own, for example:
Installing Oozie on two servers on the same partition which will be
synchronized by DRBD.
You can trigger a "failover" using linux Heartbeat and that way maintain a
virtual IP.





On Thu, Sep 1, 2011 at 1:59 PM, Per Steffensen  wrote:

> Hi
>
> Thanks a lot for pointing me to Oozie. I have looked a little bit into
> Oozie and it seems like the "component" triggering jobs is called
> "Coordinator Application". But I really see nowhere that this Coordinator
> Application doesnt just run on a single machine, and that it will therefore
> not trigger anything if this machine is down. Can you confirm that the
> "Coordinator Application"-role is distributed in a distribued Oozie setup,
> so that jobs gets triggered even if one or two machines are down?
>
> Regards, Per Steffensen
>
> Ronen Itkin skrev:
>
>  Hi
>>
>> Try to use Oozie for job coordination and work flows.
>>
>>
>>
>> On Thu, Sep 1, 2011 at 12:30 PM, Per Steffensen 
>> wrote:
>>
>>
>>
>>> Hi
>>>
>>> I use hadoop for a MapReduce job in my system. I would like to have the
>>> job
>>> run very 5th minute. Are there any "distributed" timer job stuff in
>>> hadoop?
>>> Of course I could setup a timer in an external timer framework (CRON or
>>> something like that) that invokes the MapReduce job. But CRON is only
>>> running on one particular machine, so if that machine goes down my job
>>> will
>>> not be triggered. Then I could setup the timer on all or many machines,
>>> but
>>> I would not like the job to be run in more than one instance every 5th
>>> minute, so then the timer jobs would need to coordinate who is actually
>>> starting the job "this time" and all the rest would just have to do
>>> nothing.
>>> Guess I could come up with a solution to that - e.g. writing some "lock"
>>> stuff using HDFS files or by using ZooKeeper. But I would really like if
>>> someone had already solved the problem, and provided some kind of a
>>> "distributed timer framework" running in a "cluster", so that I could
>>> just
>>> register a timer job with the cluster, and then be sure that it is
>>> invoked
>>> every 5th minute, no matter if one or two particular machines in the
>>> cluster
>>> is down.
>>>
>>> Any suggestions are very welcome.
>>>
>>> Regards, Per Steffensen
>>>
>>>
>>>
>>
>>
>>
>>
>>
>
>


-- 
*
Ronen Itkin*
Taykey | www.taykey.com