Re: namenode restart
Clients can't currently handle NN restarts gracefully. At least on my 1.x/0.20.x tests, the client didn't fail if it didn't had a new block to allocate (when the NN was being restarted, since it didn't make a connection to the NN at all) but it did fail when it was writing small blocks very fast and the NN ended the connection when it was trying to locate the next block (al)location to write to. So, don't bet on it, and instead use HA NameNodes with failover, available in 2.x onwards. On Thu, Jun 7, 2012 at 3:59 PM, Rita rmorgan...@gmail.com wrote: Running Hadoop 0.22 and I need to restart the namenode so my new rack configuration will be set into place. I am thinking of doing a quick stop and start of the namenode but what will happen to the current clients? Can they tolerate a 30 second hiccup by retrying? -- --- Get your facts first, then you can distort them as you please.-- -- Harsh J
Re: Sync and Data Replication
Mohit, On Sat, Jun 9, 2012 at 11:11 PM, Mohit Anchlia mohitanch...@gmail.com wrote: Thanks Harsh for detailed info. It clears things up. Only thing from those page is concerning is what happens when client crashes. It says you could lose upto a block worth of information. Is this still true given that NN would auto close the file? Where does it say this exactly? It is true that immediate readers will not get the last block (as it remains open and uncommitted), but once the lease recovery kicks in the file is closed successfully and the last block is indeed made available, so there's no 'data loss'. Is it a good practice to reduce NN default value so that it auto-closes before 1 hr. I've not seen people do this/need to do this. Most don't run into such a situation and it is vital to properly close() files or sync() on file streams before making it available to readers. HBase manages open files during WAL-recovery using lightweight recoverLease APIs that were added for its benefit, so it doesn't need to wait for an hour for WALs to close and recover data. -- Harsh J
Re: Sync and Data Replication
On Sun, Jun 10, 2012 at 9:39 AM, Harsh J ha...@cloudera.com wrote: Mohit, On Sat, Jun 9, 2012 at 11:11 PM, Mohit Anchlia mohitanch...@gmail.com wrote: Thanks Harsh for detailed info. It clears things up. Only thing from those page is concerning is what happens when client crashes. It says you could lose upto a block worth of information. Is this still true given that NN would auto close the file? Where does it say this exactly? It is true that immediate readers will not get the last block (as it remains open and uncommitted), but once the lease recovery kicks in the file is closed successfully and the last block is indeed made available, so there's no 'data loss'. I saw it in Coherency Model - consequences of application design paragraph. Thanks for the information. It at least helps me in that I don't have to worry about the data loss when sync is not closed. Is it a good practice to reduce NN default value so that it auto-closes before 1 hr. I've not seen people do this/need to do this. Most don't run into such a situation and it is vital to properly close() files or sync() on file streams before making it available to readers. HBase manages open files during WAL-recovery using lightweight recoverLease APIs that were added for its benefit, so it doesn't need to wait for an hour for WALs to close and recover data. -- Harsh J
