Re: Re: Is Hadoop basically not suitable for a photo archive?

2017-09-04 Thread daemeon reiydelle
Kai, this is great. It is well down the path to solving the small/object-as-file problem. Good show! *Daemeon C.M. ReiydelleSan Francisco 1.415.501.0198London 44 020 8144 9872* On Mon, Sep 4, 2017 at 8:56 PM, Zheng, Kai wrote: > A nice discussion about support of small

RE: Re: Is Hadoop basically not suitable for a photo archive?

2017-09-04 Thread Zheng, Kai
A nice discussion about support of small files in Hadoop. Not sure if this really helps, but I’d like to mention in Intel we actually has spent some time on this interesting problem domain before and again recently. We planned to develop a small files compaction optimization in the Smart

Re: Re: Is Hadoop basically not suitable for a photo archive?

2017-09-04 Thread Hayati Gonultas
I would recommend an object store such as openstack swift as another option. On Mon, Sep 4, 2017 at 1:09 PM Uwe Geercken wrote: > just my two cents: > > Maybe you can use hadoop for storing and to pack multiple files to use > hdfs in a smarter way and at the same time store

Aw: Re: Is Hadoop basically not suitable for a photo archive?

2017-09-04 Thread Uwe Geercken
just my two cents:   Maybe you can use hadoop for storing and to pack multiple files to use hdfs in a smarter way and at the same time store a limited amount of data/photos - based on time - in parallel in a different solution. I assume you won't need high performant access to the whole time

Re: Is Hadoop basically not suitable for a photo archive?

2017-09-04 Thread Alexey Eremihin
Hi Ralph, In general Hadoop is able to store such data. And even Har archives can be used with conjunction with WebHDFS (by passing offset and limit attributes). What are your reading requirements? FS meta data are not distributed and reading the data is limited by the HDFS NameNode server

Is Hadoop basically not suitable for a photo archive?

2017-09-04 Thread Ralph Soika
Hi, I know that the issue around the small-file problem was asked frequently, not only in this mailing list. I also have read already some books about Haddoop and I also started to work with Hadoop. But still I did not really understand if Hadoop is the right choice for my goals. To