Hi Keshav Adding on to others comments. You can install hive anywhere, not necessary on the namenode. You can install the same on a data node or an utility server other than name node as well, I know a few large clusters that operates so.It applies the same with pig and other libraries. You can load the data into hive table even from a remote server provided there are sufficient permissions . AFAIK, on a high level when you load data from a LOCAL INPATH (dir in lfs), what it happens is that client sends a request to namenode to copy the file into hdfs. The name node do the needful and finally the data is stored as blocks in data nodes. The namenode has a fs image (file system image) and an entry of this new file would be made there. In short the name node is involved in the process from beginning itself and it is in sync. Also in hadoop it is not the name node that communicates and syncs up with data node but the data node that syncs up with name node with periodic heart beats.
Hope it helps!... Regards Bejoy.KS ________________________________ From: Jasper Knulst <jasper.knu...@incentro.com> To: user@hive.apache.org Sent: Thursday, December 8, 2011 2:09 PM Subject: Re: Data loading from Datanode Hi Keshav, What you want is not possible I guess. You can't submit anything into HDFS without the namenode. Datanodes reports their local blocks into the namenode. If the namenode does not know them it will instruct the datanode to delete them. But whats the point? If you submit local files to HDFS (via the namenode ) from a datanode, the nn will assign to first block placement directly on that dn (provided there is room etc.) and start replicating from there. Cheers Jasper Op 8 dec. 2011 08:24 schreef "Savant, Keshav" <keshav.c.sav...@fisglobal.com> het volgende: Hi All, > >Is it possible to load data (in HDFS) using Hive Load data query from any of >the Datanode? > >So that means can we insert files into datanode directly (or from hive >installed on datanode) and then the master node syncs with datanodes later. > >Keshav C Savant >_____________ >The information contained in this message is proprietary and/or confidential. >If you are not the intended recipient, please: (i) delete the message and all >copies; (ii) do not disclose, distribute or use the message in any manner; and >(iii) notify the sender immediately. In addition, please be aware that any >message addressed to our domain is subject to archiving and review by persons >other than the intended recipient. Thank you. >