We store image/media data into second hbase cluster, but I don't see a reason why it would not work with the same cluster as a separate family for example.
-Jack On Sun, Jan 27, 2013 at 1:58 PM, yiyu jia <jia.y...@gmail.com> wrote: > Hi Jack, > > Thank you. I never heard about HOOD before. I should learn it. > > Also, do you store meta data of each video clip directly in HDFS or you > have other storage like memcache? > > thanks and regards, > > Yiyu > > > On Sun, Jan 27, 2013 at 11:56 AM, Jack Levin <magn...@gmail.com> wrote: > >> We did some experiments, open source project HOOP works well with >> interfacing to HDFS to expose REST Api interface to your file system. >> >> -Jack >> >> On Sun, Jan 27, 2013 at 7:37 AM, yiyu jia <jia.y...@gmail.com> wrote: >> > Hi Jack, >> > >> > Thanks so much for sharing! Do you have comments on storing video in >> HDFS? >> > >> > thanks and regards, >> > >> > Yiyu >> > >> > On Sat, Jan 26, 2013 at 9:56 PM, Jack Levin <magn...@gmail.com> wrote: >> > >> >> AFAIK, namenode would not like tracking 20 billion small files :) >> >> >> >> -jack >> >> >> >> On Sat, Jan 26, 2013 at 6:00 PM, S Ahmed <sahmed1...@gmail.com> wrote: >> >> > That's pretty amazing. >> >> > >> >> > What I am confused is, why did you go with hbase and not just straight >> >> into >> >> > hdfs? >> >> > >> >> > >> >> > >> >> > >> >> > On Fri, Jan 25, 2013 at 2:41 AM, Jack Levin <magn...@gmail.com> >> wrote: >> >> > >> >> >> Two people including myself, its fairly hands off. Took about 3 >> months >> >> to >> >> >> tune it right, however we did have had multiple years of experience >> with >> >> >> datanodes and hadoop in general, so that was a good boost. >> >> >> >> >> >> We have 4 hbase clusters today, image store being largest >> >> >> On Jan 24, 2013 2:14 PM, "S Ahmed" <sahmed1...@gmail.com> wrote: >> >> >> >> >> >> > Jack, out of curiosity, how many people manage the hbase related >> >> servers? >> >> >> > >> >> >> > Does it require constant monitoring or its fairly hands-off now? >> (or >> >> a >> >> >> bit >> >> >> > of both, early days was getting things write/learning and now its >> >> purring >> >> >> > along). >> >> >> > >> >> >> > >> >> >> > On Wed, Jan 23, 2013 at 11:53 PM, Jack Levin <magn...@gmail.com> >> >> wrote: >> >> >> > >> >> >> > > Its best to keep some RAM for caching of the filesystem, besides >> we >> >> >> > > also run datanode which takes heap as well. >> >> >> > > Now, please keep in mind that even if you specify heap of say >> 5GB, >> >> if >> >> >> > > your server opens threads to communicate with other systems via >> RPC >> >> >> > > (which hbase does a lot), you will indeed use HEAP + >> >> >> > > Nthreads*thread*kb_size. There is a good Sun Microsystems >> document >> >> >> > > about it. (I don't have the link handy). >> >> >> > > >> >> >> > > -Jack >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > On Mon, Jan 21, 2013 at 5:10 PM, Varun Sharma < >> va...@pinterest.com> >> >> >> > wrote: >> >> >> > > > Thanks for the useful information. I wonder why you use only 5G >> >> heap >> >> >> > when >> >> >> > > > you have an 8G machine ? Is there a reason to not use all of it >> >> (the >> >> >> > > > DataNode typically takes a 1G of RAM) >> >> >> > > > >> >> >> > > > On Sun, Jan 20, 2013 at 11:49 AM, Jack Levin < >> magn...@gmail.com> >> >> >> > wrote: >> >> >> > > > >> >> >> > > >> I forgot to mention that I also have this setup: >> >> >> > > >> >> >> >> > > >> <property> >> >> >> > > >> <name>hbase.hregion.memstore.flush.size</name> >> >> >> > > >> <value>33554432</value> >> >> >> > > >> <description>Flush more often. Default: >> 67108864</description> >> >> >> > > >> </property> >> >> >> > > >> >> >> >> > > >> This parameter works on per region amount, so this means if >> any >> >> of >> >> >> my >> >> >> > > >> 400 (currently) regions on a regionserver has 30MB+ in >> memstore, >> >> the >> >> >> > > >> hbase will flush it to disk. >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> Here are some metrics from a regionserver: >> >> >> > > >> >> >> >> > > >> requests=2, regions=370, stores=370, storefiles=1390, >> >> >> > > >> storefileIndexSize=304, memstoreSize=2233, >> compactionQueueSize=0, >> >> >> > > >> flushQueueSize=0, usedHeap=3516, maxHeap=4987, >> >> >> > > >> blockCacheSize=790656256, blockCacheFree=255245888, >> >> >> > > >> blockCacheCount=2436, blockCacheHitCount=218015828, >> >> >> > > >> blockCacheMissCount=13514652, blockCacheEvictedCount=2561516, >> >> >> > > >> blockCacheHitRatio=94, blockCacheHitCachingRatio=98 >> >> >> > > >> >> >> >> > > >> Note, that memstore is only 2G, this particular regionserver >> >> HEAP is >> >> >> > set >> >> >> > > >> to 5G. >> >> >> > > >> >> >> >> > > >> And last but not least, its very important to have good GC >> setup: >> >> >> > > >> >> >> >> > > >> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -Xms5000m >> >> >> > > >> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails >> >> >> > > >> -XX:+PrintGCDateStamps >> >> >> > > >> -XX:+HeapDumpOnOutOfMemoryError >> >> >> -Xloggc:$HBASE_HOME/logs/gc-hbase.log >> >> >> > \ >> >> >> > > >> -XX:MaxTenuringThreshold=15 -XX:SurvivorRatio=8 \ >> >> >> > > >> -XX:+UseParNewGC \ >> >> >> > > >> -XX:NewSize=128m -XX:MaxNewSize=128m \ >> >> >> > > >> -XX:-UseAdaptiveSizePolicy \ >> >> >> > > >> -XX:+CMSParallelRemarkEnabled \ >> >> >> > > >> -XX:-TraceClassUnloading >> >> >> > > >> " >> >> >> > > >> >> >> >> > > >> -Jack >> >> >> > > >> >> >> >> > > >> On Thu, Jan 17, 2013 at 3:29 PM, Varun Sharma < >> >> va...@pinterest.com> >> >> >> > > wrote: >> >> >> > > >> > Hey Jack, >> >> >> > > >> > >> >> >> > > >> > Thanks for the useful information. By flush size being 15 >> %, do >> >> >> you >> >> >> > > mean >> >> >> > > >> > the memstore flush size ? 15 % would mean close to 1G, have >> you >> >> >> seen >> >> >> > > any >> >> >> > > >> > issues with flushes taking too long ? >> >> >> > > >> > >> >> >> > > >> > Thanks >> >> >> > > >> > Varun >> >> >> > > >> > >> >> >> > > >> > On Sun, Jan 13, 2013 at 8:17 AM, Jack Levin < >> magn...@gmail.com >> >> > >> >> >> > > wrote: >> >> >> > > >> > >> >> >> > > >> >> That's right, Memstore size , not flush size is increased. >> >> >> > Filesize >> >> >> > > is >> >> >> > > >> >> 10G. Overall write cache is 60% of heap and read cache is >> 20%. >> >> >> > Flush >> >> >> > > >> size >> >> >> > > >> >> is 15%. 64 maxlogs at 128MB. One namenode server, one >> >> secondary >> >> >> > that >> >> >> > > >> can >> >> >> > > >> >> be promoted. On the way to hbase images are written to a >> >> queue, >> >> >> so >> >> >> > > >> that we >> >> >> > > >> >> can take Hbase down for maintenance and still do inserts >> >> later. >> >> >> > > >> ImageShack >> >> >> > > >> >> has ‘perma cache’ servers that allows writes and serving of >> >> data >> >> >> > even >> >> >> > > >> when >> >> >> > > >> >> hbase is down for hours, consider it 4th replica 😉 >> outside of >> >> >> > hadoop >> >> >> > > >> >> >> >> >> > > >> >> Jack >> >> >> > > >> >> >> >> >> > > >> >> *From:* Mohit Anchlia <mohitanch...@gmail.com> >> >> >> > > >> >> *Sent:* January 13, 2013 7:48 AM >> >> >> > > >> >> *To:* user@hbase.apache.org >> >> >> > > >> >> *Subject:* Re: Storing images in Hbase >> >> >> > > >> >> >> >> >> > > >> >> Thanks Jack for sharing this information. This definitely >> >> makes >> >> >> > sense >> >> >> > > >> when >> >> >> > > >> >> using the type of caching layer. You mentioned about >> >> increasing >> >> >> > write >> >> >> > > >> >> cache, I am assuming you had to increase the following >> >> parameters >> >> >> > in >> >> >> > > >> >> addition to increase the memstore size: >> >> >> > > >> >> >> >> >> > > >> >> hbase.hregion.max.filesize >> >> >> > > >> >> hbase.hregion.memstore.flush.size >> >> >> > > >> >> >> >> >> > > >> >> On Fri, Jan 11, 2013 at 9:47 AM, Jack Levin < >> >> magn...@gmail.com> >> >> >> > > wrote: >> >> >> > > >> >> >> >> >> > > >> >> > We buffer all accesses to HBASE with Varnish SSD based >> >> caching >> >> >> > > layer. >> >> >> > > >> >> > So the impact for reads is negligible. We have 70 node >> >> >> cluster, >> >> >> > 8 >> >> >> > > GB >> >> >> > > >> >> > of RAM per node, relatively weak nodes (intel core 2 >> duo), >> >> with >> >> >> > > >> >> > 10-12TB per server of disks. Inserting 600,000 images >> per >> >> day. >> >> >> > We >> >> >> > > >> >> > have relatively little of compaction activity as we made >> our >> >> >> > write >> >> >> > > >> >> > cache much larger than read cache - so we don't >> experience >> >> >> region >> >> >> > > file >> >> >> > > >> >> > fragmentation as much. >> >> >> > > >> >> > >> >> >> > > >> >> > -Jack >> >> >> > > >> >> > >> >> >> > > >> >> > On Fri, Jan 11, 2013 at 9:40 AM, Mohit Anchlia < >> >> >> > > >> mohitanch...@gmail.com> >> >> >> > > >> >> > wrote: >> >> >> > > >> >> > > I think it really depends on volume of the traffic, >> data >> >> >> > > >> distribution >> >> >> > > >> >> per >> >> >> > > >> >> > > region, how and when files compaction occurs, number of >> >> nodes >> >> >> > in >> >> >> > > the >> >> >> > > >> >> > > cluster. In my experience when it comes to blob data >> where >> >> >> you >> >> >> > > are >> >> >> > > >> >> > serving >> >> >> > > >> >> > > 10s of thousand+ requests/sec writes and reads then >> it's >> >> very >> >> >> > > >> difficult >> >> >> > > >> >> > to >> >> >> > > >> >> > > manage HBase without very hard operations and >> maintenance >> >> in >> >> >> > > play. >> >> >> > > >> Jack >> >> >> > > >> >> > > earlier mentioned they have 1 billion images, It would >> be >> >> >> > > >> interesting >> >> >> > > >> >> to >> >> >> > > >> >> > > know what they see in terms of compaction, no of >> requests >> >> per >> >> >> > > sec. >> >> >> > > >> I'd >> >> >> > > >> >> be >> >> >> > > >> >> > > surprised that in high volume site it can be done >> without >> >> any >> >> >> > > >> Caching >> >> >> > > >> >> > layer >> >> >> > > >> >> > > on the top to alleviate IO spikes that occurs because >> of >> >> GC >> >> >> and >> >> >> > > >> >> > compactions. >> >> >> > > >> >> > > >> >> >> > > >> >> > > On Fri, Jan 11, 2013 at 7:27 AM, Mohammad Tariq < >> >> >> > > donta...@gmail.com >> >> >> > > >> > >> >> >> > > >> >> > wrote: >> >> >> > > >> >> > > >> >> >> > > >> >> > >> IMHO, if the image files are not too huge, Hbase can >> >> >> > efficiently >> >> >> > > >> serve >> >> >> > > >> >> > the >> >> >> > > >> >> > >> purpose. You can store some additional info along with >> >> the >> >> >> > file >> >> >> > > >> >> > depending >> >> >> > > >> >> > >> upon your search criteria to make the search faster. >> Say >> >> if >> >> >> > you >> >> >> > > >> want >> >> >> > > >> >> to >> >> >> > > >> >> > >> fetch images by the type, you can store images in one >> >> column >> >> >> > and >> >> >> > > >> its >> >> >> > > >> >> > >> extension in another column(jpg, tiff etc). >> >> >> > > >> >> > >> >> >> >> > > >> >> > >> BTW, what exactly is the problem which you are facing. >> >> You >> >> >> > have >> >> >> > > >> >> written >> >> >> > > >> >> > >> "But I still cant do it"? >> >> >> > > >> >> > >> >> >> >> > > >> >> > >> Warm Regards, >> >> >> > > >> >> > >> Tariq >> >> >> > > >> >> > >> https://mtariq.jux.com/ >> >> >> > > >> >> > >> >> >> >> > > >> >> > >> >> >> >> > > >> >> > >> On Fri, Jan 11, 2013 at 8:30 PM, Michael Segel < >> >> >> > > >> >> > michael_se...@hotmail.com >> >> >> > > >> >> > >> >wrote: >> >> >> > > >> >> > >> >> >> >> > > >> >> > >> > That's a viable option. >> >> >> > > >> >> > >> > HDFS reads are faster than HBase, but it would >> require >> >> >> first >> >> >> > > >> hitting >> >> >> > > >> >> > the >> >> >> > > >> >> > >> > index in HBase which points to the file and then >> >> fetching >> >> >> > the >> >> >> > > >> file. >> >> >> > > >> >> > >> > It could be faster... we found storing binary data >> in a >> >> >> > > sequence >> >> >> > > >> >> file >> >> >> > > >> >> > and >> >> >> > > >> >> > >> > indexed on HBase to be faster than HBase, however, >> YMMV >> >> >> and >> >> >> > > HBase >> >> >> > > >> >> has >> >> >> > > >> >> > >> been >> >> >> > > >> >> > >> > improved since we did that project.... >> >> >> > > >> >> > >> > >> >> >> > > >> >> > >> > >> >> >> > > >> >> > >> > On Jan 10, 2013, at 10:56 PM, shashwat shriparv < >> >> >> > > >> >> > >> dwivedishash...@gmail.com> >> >> >> > > >> >> > >> > wrote: >> >> >> > > >> >> > >> > >> >> >> > > >> >> > >> > > Hi Kavish, >> >> >> > > >> >> > >> > > >> >> >> > > >> >> > >> > > i have a better idea for you copy your image files >> >> to a >> >> >> > > single >> >> >> > > >> >> file >> >> >> > > >> >> > on >> >> >> > > >> >> > >> > > hdfs, and if new image comes append it to the >> >> existing >> >> >> > > image, >> >> >> > > >> and >> >> >> > > >> >> > keep >> >> >> > > >> >> > >> > and >> >> >> > > >> >> > >> > > update the metadata and the offset to the HBase. >> >> Because >> >> >> > if >> >> >> > > you >> >> >> > > >> >> put >> >> >> > > >> >> > >> > bigger >> >> >> > > >> >> > >> > > image in hbase it wil lead to some issue. >> >> >> > > >> >> > >> > > >> >> >> > > >> >> > >> > > >> >> >> > > >> >> > >> > > >> >> >> > > >> >> > >> > > ∞ >> >> >> > > >> >> > >> > > Shashwat Shriparv >> >> >> > > >> >> > >> > > >> >> >> > > >> >> > >> > > >> >> >> > > >> >> > >> > > >> >> >> > > >> >> > >> > > On Fri, Jan 11, 2013 at 9:21 AM, lars hofhansl < >> >> >> > > >> la...@apache.org> >> >> >> > > >> >> > >> wrote: >> >> >> > > >> >> > >> > > >> >> >> > > >> >> > >> > >> Interesting. That's close to a PB if my math is >> >> >> correct. >> >> >> > > >> >> > >> > >> Is there a write up about this somewhere? >> Something >> >> >> that >> >> >> > we >> >> >> > > >> could >> >> >> > > >> >> > link >> >> >> > > >> >> > >> > >> from the HBase homepage? >> >> >> > > >> >> > >> > >> >> >> >> > > >> >> > >> > >> -- Lars >> >> >> > > >> >> > >> > >> >> >> >> > > >> >> > >> > >> >> >> >> > > >> >> > >> > >> ----- Original Message ----- >> >> >> > > >> >> > >> > >> From: Jack Levin <magn...@gmail.com> >> >> >> > > >> >> > >> > >> To: user@hbase.apache.org >> >> >> > > >> >> > >> > >> Cc: Andrew Purtell <apurt...@apache.org> >> >> >> > > >> >> > >> > >> Sent: Thursday, January 10, 2013 9:24 AM >> >> >> > > >> >> > >> > >> Subject: Re: Storing images in Hbase >> >> >> > > >> >> > >> > >> >> >> >> > > >> >> > >> > >> We stored about 1 billion images into hbase with >> >> file >> >> >> > size >> >> >> > > up >> >> >> > > >> to >> >> >> > > >> >> > 10MB. >> >> >> > > >> >> > >> > >> Its been running for close to 2 years without >> issues >> >> >> and >> >> >> > > >> serves >> >> >> > > >> >> > >> > >> delivery of images for Yfrog and ImageShack. If >> you >> >> >> have >> >> >> > > any >> >> >> > > >> >> > >> > >> questions about the setup, I would be glad to >> answer >> >> >> > them. >> >> >> > > >> >> > >> > >> >> >> >> > > >> >> > >> > >> -Jack >> >> >> > > >> >> > >> > >> >> >> >> > > >> >> > >> > >> On Sun, Jan 6, 2013 at 1:09 PM, Mohit Anchlia < >> >> >> > > >> >> > mohitanch...@gmail.com >> >> >> > > >> >> > >> > >> >> >> > > >> >> > >> > >> wrote: >> >> >> > > >> >> > >> > >>> I have done extensive testing and have found >> that >> >> >> blobs >> >> >> > > don't >> >> >> > > >> >> > belong >> >> >> > > >> >> > >> in >> >> >> > > >> >> > >> > >> the >> >> >> > > >> >> > >> > >>> databases but are rather best left out on the >> file >> >> >> > system. >> >> >> > > >> >> Andrew >> >> >> > > >> >> > >> > >> outlined >> >> >> > > >> >> > >> > >>> issues that you'll face and not to mention IO >> >> issues >> >> >> > when >> >> >> > > >> >> > compaction >> >> >> > > >> >> > >> > >> occurs >> >> >> > > >> >> > >> > >>> over large files. >> >> >> > > >> >> > >> > >>> >> >> >> > > >> >> > >> > >>> On Sun, Jan 6, 2013 at 12:52 PM, Andrew Purtell >> < >> >> >> > > >> >> > apurt...@apache.org >> >> >> > > >> >> > >> > >> >> >> > > >> >> > >> > >> wrote: >> >> >> > > >> >> > >> > >>> >> >> >> > > >> >> > >> > >>>> I meant this to say "a few really large values" >> >> >> > > >> >> > >> > >>>> >> >> >> > > >> >> > >> > >>>> On Sun, Jan 6, 2013 at 12:49 PM, Andrew >> Purtell < >> >> >> > > >> >> > >> apurt...@apache.org> >> >> >> > > >> >> > >> > >>>> wrote: >> >> >> > > >> >> > >> > >>>> >> >> >> > > >> >> > >> > >>>>> Consider if the split threshold is 2 GB but >> your >> >> one >> >> >> > row >> >> >> > > >> >> > contains >> >> >> > > >> >> > >> 10 >> >> >> > > >> >> > >> > >> GB >> >> >> > > >> >> > >> > >>>> as >> >> >> > > >> >> > >> > >>>>> really large value. >> >> >> > > >> >> > >> > >>>> >> >> >> > > >> >> > >> > >>>> >> >> >> > > >> >> > >> > >>>> >> >> >> > > >> >> > >> > >>>> >> >> >> > > >> >> > >> > >>>> -- >> >> >> > > >> >> > >> > >>>> Best regards, >> >> >> > > >> >> > >> > >>>> >> >> >> > > >> >> > >> > >>>> - Andy >> >> >> > > >> >> > >> > >>>> >> >> >> > > >> >> > >> > >>>> Problems worthy of attack prove their worth by >> >> >> hitting >> >> >> > > >> back. - >> >> >> > > >> >> > Piet >> >> >> > > >> >> > >> > Hein >> >> >> > > >> >> > >> > >>>> (via Tom White) >> >> >> > > >> >> > >> > >>>> >> >> >> > > >> >> > >> > >> >> >> >> > > >> >> > >> > >> >> >> >> > > >> >> > >> > >> >> >> > > >> >> > >> > >> >> >> > > >> >> > >> >> >> >> > > >> >> > >> >> >> > > >> >> >> >> >> > > >> >> >> >> > > >> >> >> > >> >> >> >> >> >>
