Kai, this is great. It is well down the path to solving the
small/object-as-file problem. Good show!
*Daemeon C.M. ReiydelleSan Francisco 1.415.501.0198London 44 020 8144 9872*
On Mon, Sep 4, 2017 at 8:56 PM, Zheng, Kai wrote:
> A nice discussion about support of small
A nice discussion about support of small files in Hadoop.
Not sure if this really helps, but I’d like to mention in Intel we actually has
spent some time on this interesting problem domain before and again recently.
We planned to develop a small files compaction optimization in the Smart
I would recommend an object store such as openstack swift as another option.
On Mon, Sep 4, 2017 at 1:09 PM Uwe Geercken wrote:
> just my two cents:
>
> Maybe you can use hadoop for storing and to pack multiple files to use
> hdfs in a smarter way and at the same time store
just my two cents:
Maybe you can use hadoop for storing and to pack multiple files to use hdfs in a smarter way and at the same time store a limited amount of data/photos - based on time - in parallel in a different solution. I assume you won't need high performant access to the whole time
Hi Ralph,
In general Hadoop is able to store such data. And even Har archives can be
used with conjunction with WebHDFS (by passing offset and limit
attributes). What are your reading requirements? FS meta data are not
distributed and reading the data is limited by the HDFS NameNode server
Hi,
I know that the issue around the small-file problem was asked
frequently, not only in this mailing list.
I also have read already some books about Haddoop and I also started to
work with Hadoop. But still I did not really understand if Hadoop is the
right choice for my goals.
To