I've diagramed the hadoop HDFS write path here: http://jayunit100.blogspot.com/2013/04/the-kv-pair-salmon-run-in-mapreduce-hdfs.html
On Tue, Oct 1, 2013 at 5:24 PM, Ravi Prakash <ravi...@ymail.com> wrote: > Karim! > > Look at DFSOutputStream.java:DataStreamer > > HTH > Ravi > > > ------------------------------ > *From:* Karim Awara <karim.aw...@kaust.edu.sa> > *To:* user <user@hadoop.apache.org> > *Sent:* Thursday, September 26, 2013 7:51 AM > *Subject:* Re: Uploading a file to HDFS > > > Thanks for the reply. when the client caches 64KB of data on its own side, > do you know which set of major java classes/files responsible for such > action? > > -- > Best Regards, > Karim Ahmed Awara > > > On Thu, Sep 26, 2013 at 2:25 PM, Jitendra Yadav < > jeetuyadav200...@gmail.com> wrote: > > Case 2: > > While selecting target DN in case of write operations, NN will always > prefers first DN as same DN from where client sending the data, in some > cases NN ignore that DN when there is some disk space issues or some other > health symptoms found,rest of things will same. > > Thanks > Jitendra > > > On Thu, Sep 26, 2013 at 4:15 PM, Shekhar Sharma <shekhar2...@gmail.com>wrote: > > Its not the namenode that does the reading or breaking of the file.. > When you run the command hadoop fs -put <input> <output>..... > Here "hadoop" is a script file which is default client for hadoop..and > when the client contacts the namenode for writing, then NN creates a > block id and ask 3 dN to host the block ( replication factor to 3) and > this information is sent to client. > > client caches 64KB of data on its own side and then pushes the data to > the DN and then this data gets pushed through pipeline..and this > process gets repeated till 64MB data is written and if the client > wants to to write more then he will again contact NN via heart beat > signal and this process continuess... > > Check how does writing happens in HDFS? > > > Regards, > Som Shekhar Sharma > +91-8197243810 > > > On Thu, Sep 26, 2013 at 3:41 PM, Karim Awara <karim.aw...@kaust.edu.sa> > wrote: > > Hi, > > > > I have a couple of questions about the process of uploading a large file > (> > > 10GB) to HDFS. > > > > To make sure my understanding is correct, assuming I have a cluster of N > > machines. > > > > What happens in the following: > > > > > > Case 1: > > assuming i want to uppload a file (input.txt) of size K > GBs > > that resides on the local disk of machine 1 (which happens to be the > > namenode only). if I am running the command -put input.txt {some hdfs > dir} > > from the namenode (assuming it does not play the datanode role), then > will > > the namenode read the first 64MB in a temporary pipe and then transfers > it > > to one of the cluster datanodes once finished? Or the namenode does not > do > > any reading of the file, but rather asks a certain datanode to read the > 64MB > > window from the file remotely? > > > > > > Case 2: > > assume machine 1 is the namenode, but i run the -put command > > from machine 3 (which is a datanode). who will start reading the file? > > > > > > > > -- > > Best Regards, > > Karim Ahmed Awara > > > > ________________________________ > > This message and its contents, including attachments are intended solely > for > > the original recipient. If you are not the intended recipient or have > > received this message in error, please notify me immediately and delete > this > > message from your computer system. Any unauthorized use or distribution > is > > prohibited. Please consider the environment before printing this email. > > > > > ------------------------------ > This message and its contents, including attachments are intended solely > for the original recipient. If you are not the intended recipient or have > received this message in error, please notify me immediately and delete > this message from your computer system. Any unauthorized use or > distribution is prohibited. Please consider the environment before printing > this email. > > > -- Jay Vyas http://jayunit100.blogspot.com