Sorry for the inconvenience caused ...I will not spam core dev. Scale we are thinking in terms of more nodes in coming future can go to petabytes of data Can you please give some pointers for handling the same issue, i am quite new to hadoop
Regards, Snehal Raghu Angadi wrote: > > > What is scale you are thinking of? (10s, 100s or more nodes)? > > The memory for metadata at NameNode you mentioned is that main issue > with small files. There are multiple alternatives for the dealing with > that. This issue is discussed many times here. > > Also please use core-user@ id alone for asking for help.. you don't need > to send to core-devel@ > > Raghu. > > snehal nagmote wrote: >> Hello Sir, >> >> I have some doubts, please help me. >> we have requirement of scalable storage system, we have developed one >> agro-advisory system in which farmers will sent the crop pictures >> particularly in sequential manner some >> 6-7 photos of 3-4 kb each would be stored in storage server and these >> photos >> would be read sequentially by scientist to detect the problem, writing to >> images would not be done. >> >> So for storing these images we are using hadoop file system, is it >> feasible >> to use hadoop >> file system for the same purpose. >> >> As also the images are of only 3-4 kb and hadoop reads the data in blocks >> of >> size 64 mb >> how can we increase the performance, what could be the tricks and tweaks >> that should be done to use hadoop for such kind of purpose. >> >> Next problem is as hadoop stores all the metadata in memory,can we use >> some >> mechanism to store the files in the block of some greater size because as >> the files would be of small size,so it will store the lots metadata and >> will >> overflow the main memory >> please suggest what could be done >> >> >> regards, >> Snehal >> > > > -- View this message in context: http://www.nabble.com/hadoop-need-help-please-suggest-tp22666530p22721718.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
