Hi Keshav
Adding on to others comments. You can install hive anywhere, not
necessary on the namenode. You can install the same on a data node or an
utility server other than name node as well, I know a few large clusters that
operates so.It applies the same with pig and other libraries. You can load the
data into hive table even from a remote server provided there are sufficient
permissions .
AFAIK, on a high level when you load data from a LOCAL INPATH (dir
in lfs), what it happens is that client sends a request to namenode to copy the
file into hdfs. The name node do the needful and finally the data is stored as
blocks in data nodes. The namenode has a fs image (file system image) and an
entry of this new file would be made there. In short the name node is involved
in the process from beginning itself and it is in sync. Also in hadoop it is
not the name node that communicates and syncs up with data node but the data
node that syncs up with name node with periodic heart beats.
Hope it helps!...
Regards
Bejoy.KS
From: Jasper Knulst jasper.knu...@incentro.com
To: user@hive.apache.org
Sent: Thursday, December 8, 2011 2:09 PM
Subject: Re: Data loading from Datanode
Hi Keshav,
What you want is not possible I guess. You can't submit anything into HDFS
without the namenode. Datanodes reports their local blocks into the namenode.
If the namenode does not know them it will instruct the datanode to delete
them.
But whats the point? If you submit local files to HDFS (via the namenode ) from
a datanode, the nn will assign to first block placement directly on that dn
(provided there is room etc.) and start replicating from there.
Cheers Jasper
Op 8 dec. 2011 08:24 schreef Savant, Keshav keshav.c.sav...@fisglobal.com
het volgende:
Hi All,
Is it possible to load data (in HDFS) using Hive Load data query from any of
the Datanode?
So that means can we insert files into datanode directly (or from hive
installed on datanode) and then the master node syncs with datanodes later.
Keshav C Savant
_
The information contained in this message is proprietary and/or confidential.
If you are not the intended recipient, please: (i) delete the message and all
copies; (ii) do not disclose, distribute or use the message in any manner; and
(iii) notify the sender immediately. In addition, please be aware that any
message addressed to our domain is subject to archiving and review by persons
other than the intended recipient. Thank you.
