Re: Store Large files on HBase/HDFS

On Thu, Mar 31, 2016 at 6:42 PM, Arun Patel wrote: > [image: Mic Drop] > Since there are millions of files (with sizes from 1mb to 15mb), I would > like to store them in a sequence file. How do I store the location of each > of these files in HBase? > > I see lots blogs

Re: Store Large files on HBase/HDFS

[image: Mic Drop] Since there are millions of files (with sizes from 1mb to 15mb), I would like to store them in a sequence file. How do I store the location of each of these files in HBase? I see lots blogs and books talking about storing large files on HDFS and storing file paths on HBase.

Re: Store Large files on HBase/HDFS

For #1, please take a look at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java e.g. the following methods: public DFSInputStream open(String src) throws IOException { public HdfsDataOutputStream append(final String src, final int buffersize,

Re: Store Large files on HBase/HDFS

But the whole idea of storing large file on HDFS will be defeated, right? Why do you think we need to bring it back to HBase? On Thu, Feb 18, 2016 at 10:23 PM, Jameson Li wrote: > maybe U can parse the HDFS image file, then transform them as the Hfile, > and load into hbase

Re: Store Large files on HBase/HDFS

maybe U can parse the HDFS image file, then transform them as the Hfile, and load into hbase Tables. --remember to partition the hbase table 2016-02-18 7:40 GMT+08:00 Arun Patel : > I would like to store large documents (over 100 MB) on HDFS and insert > metadata in

Store Large files on HBase/HDFS

I would like to store large documents (over 100 MB) on HDFS and insert metadata in HBase. 1) Users will use HBase REST API for PUT and GET requests for storing and retrieving documents. In this case, how to PUT and GET documents to/from HDFS?What are the recommended ways for storing and accessing