Thanks Andy. I believe that the 0.20+ will be great improved.
On Mon, Jun 15, 2009 at 12:40 AM, Andrew Purtell <[email protected]>wrote: > Hi Schubert, > > I have 2TB and 1TB storage densities, respectively, on my test > environments, > so I very much understand your point of view. > > I think 0.20 will be able to come a lot closer to the goal of utilizing all > of that space than 0.19 can. If you can wait for the release of 0.20, it > may > be worth experimenting to try and achieve 2000 regions with 1GB store > files. > I personally am planning to run such an experiment. > > - Andy > > > > > ________________________________ > From: zsongbo <[email protected]> > To: [email protected] > Sent: Sunday, June 14, 2009 9:34:44 AM > Subject: Re: Help needed - Adding HBase to architecture > > Thank Andy and stack for your experiences sharing.In my data management > system with HBase, I want to store HUGE size of data. > But, assumes each node servers 1000 regions (250MB), only 250GB storage are > used. We have 2TB disk on each node. > So, now, we store the data in files in HDFS and create a simple index to > query and locate these files. > > Schubert > > On Sun, Jun 14, 2009 at 3:18 AM, Andrew Purtell <[email protected]> > wrote: > > > 0.19 will have trouble compacting regions with large store files (> 1GB), > > especially if they are compressed. > > > > 0.20 is such a game changer that all the old experience and assumptions > > will have to be thrown out and all of this testing redone. That is a very > > good thing! :-) Kudos to all those who rebuilt the region server for this > > release. > > > > - Andy > > > > > > > > > > ________________________________ > > From: stack <[email protected]> > > To: [email protected] > > Sent: Saturday, June 13, 2009 12:13:58 PM > > Subject: Re: Help needed - Adding HBase to architecture > > > > At powerset, we have ~80 regions per node on > 100 nodes. > > > > I've seen other clusters with hundreds and in testing have come close to > a > > thousand per node. > > > > When a node has this many regions on board and it crashes, its going to > > take > > a while to recover. > > > > We've not played with it in a while but regions could be fatter. By > > default, biggest store file in a region is < 256M. Dependent on the type > > of > > your data and your access patterns, we should probably look to doubling > or > > quadrupling this size. Then could carry low hundreds of regions but > they'd > > have more heft to them. > > > > St.Ack > > > > On Sat, Jun 13, 2009 at 11:59 AM, zsongbo <[email protected]> wrote: > > > > > Hi Billy, > > > > > > I agree "Hbase would be better suited to store the meta data in place > of > > > the > > > images." very much.And store files in HDFS or other storage system such > > as > > > S3. But for small files, S3-like object storage system will be better. > > > > > > Another issue to discuss with you: > > > How many tablets/regions served in each of you HBase region server in > you > > > practices? The Bigtable paper suggests at most handreds. > > > > > > Schubert > > > > > > On Mon, Jun 8, 2009 at 2:28 PM, Billy Pearson < > > [email protected] > > > >wrote: > > > > > > > If I was going to use a RDBMS to store the meta data then I would > just > > > use > > > > hadoop hdfs to store the images/video > > > > I know that hadoop has a thrift api now > > > > http://wiki.apache.org/hadoop/HDFS-APIs > > > > > > > > Hbase would be better suited to store the meta data in place of the > > > images. > > > > The biggest benefit to hbase is you can scale the reads and writes to > > the > > > > db not just the reads in most RDBMS > > > > > > > > So you should be able to work with the files in hadoop in any > language > > as > > > > long as you can get hadoop working correctly on windows. > > > > The benefit of this is you can scale hadoop as needed to hold more > > data. > > > > The downside to this is the memory that will be required for the > > namenode > > > > I thank its like 3m files per gb of memory or something like that > > > > > > > > > > > > > > > > > > > > > > > > "Nitin Gupta" <[email protected]> wrote in message > > > > news:003c01c9e7fc$2087df00$61979d...@com... > > > > > > > > Jonathan, > > > >> > > > >> Thanks for detailed explanation. Much helpful. > > > >> > > > >> As far as file size is concerned, we may be even required to save > > Videos > > > >> in > > > >> future. So we shall def go above the HBase size limit at some point > in > > > >> time. > > > >> Any other solution or key-value database that you can recommend for > > our > > > >> case? > > > >> > > > >> I am not much knowledgeable about the HDFS either. I think if we go > > with > > > >> pure HDFS, then all the required DB operations would have to be > custom > > > >> developed on top of HDFS. For our needs, do you think that HDFS > > already > > > >> has > > > >> enough support that we will not need any major custom development. > We > > > are > > > >> just saving the files/attachements and retrieving them with some > basic > > > >> search. > > > >> > > > >> Regards, > > > >> Nitin > > > >> > > > >> -----Original Message----- > > > >> From: Jonathan Gray [mailto:[email protected]] > > > >> Sent: Sunday, June 07, 2009 9:30 PM > > > >> To: [email protected] > > > >> Subject: Re: Help needed - Adding HBase to architecture > > > >> > > > >> Nitin, > > > >> > > > >> HBase stores arbitrary binary values (row keys, column qualifiers, > and > > > >> column values), so it is certainly capable of storing and serving > > files > > > >> and images. > > > >> > > > >> My only real question before I would give you a +1 on your idea is > > what > > > >> you expect the range of file sizes to be. While HBase allows you to > > > store > > > >> values up to length Integer.MAX_VALUE, that is not recommended and > in > > > past > > > >> versions has lead to memory issues (OOME and such). > > > >> > > > >> Images, text, word/excel docs, etc... should be no problem. But I > > don't > > > >> recommend storing things in the upper 10s or 100s of MB, though it's > > > >> probably possible with a little work adjusting some configuration > > > >> parameters. In general, if you are approaching HDFS block size, > then > > > you > > > >> really just want HDFS and not HBase :) > > > >> > > > >> We are not currently running this in production, but we have had an > > > >> experimental version of our media server that runs on top of HBase > > > rather > > > >> than the file system. It has a series of Python scripts (connected > to > > > >> HBase through our custom interface, you could use Java directly or > > > >> Thrift/REST/etc) that are responsible for generating various > thumbnail > > > >> sizes. The originals are stored in HBase, and then a special query > is > > > run > > > >> to grab the thumbnail of a certain size. If it exists in HBase > > already, > > > >> it is just fetched and returned. Otherwise, it is generated (via > PIL, > > > >> Python Imaging Library, and some other custom tools), stored in > HBase, > > > and > > > >> then returned to the client. > > > >> > > > >> As far as HBase on Windows goes... It's currently not possible but > > there > > > >> has been some effort from Powerset/Microsoft to make it happen. I > > will > > > >> yield to those more familiar with it. > > > >> > > > >> Personally, I run Windows on my primary work desktop and spend a > good > > > >> chunk of my time on HBase development. When I've wanted to spin up > > > >> pseudo-distributed local clusters, I usually use a cheap Linux node > or > > > >> local Virtual Machine. In both cases, I use a Windows X Server and > > > >> redirect output to my local Windows machine so I can run Eclipse and > > > unit > > > >> tests from my Windows GUI. Others have used Cygwin with some > success, > > I > > > >> believe. > > > >> > > > >> Hope that sheds some light for you. > > > >> > > > >> You are almost certainly right about not wanting to store this in an > > > >> RDBMS. And a hybrid approach seems to make sense, especially as a > > first > > > >> step. > > > >> > > > >> Jonathan Gray > > > >> > > > >> > > > >> > > > >> On Sun, June 7, 2009 6:44 am, Nitin Gupta wrote: > > > >> > > > >>> Hi All, > > > >>> I am working on an application which is kind of a social network on > > > >>> mobile > > > >>> WAP. Recently, we have incorporated the files or attachments > support > > > in > > > >>> our application. Right now, since we are not in production yet, we > > are > > > >>> keeping all the files in the RDBMS which our application is using. > > But > > > I > > > >>> am more than convinvced that this is not going to work once we are > in > > > >>> production mode. > > > >>> > > > >>> I got to know about HBase and I am making myself convice about its > > > usage > > > >>> for the file storage, search and retrieval operations. I would like > > my > > > >>> opinion to be endorsed by expert HBase users/developers. Just for > the > > > >>> clarification, here is what I am planning to do: > > > >>> > > > >>> Make use of a RDBMS for relational data in the application. > > > >>> All the files/blob data to be saved in the HBase. > > > >>> When required, my application can query app data from the RDBMS and > > the > > > >>> files can be retrieved from the HBase data store I will keep the > meta > > > >>> data > > > >>> of the files in my rdbms so that files can be associated with my > apps > > > >>> entities > > > >>> > > > >>> Please help me decide if this is the right approach. My app is > > supposed > > > >>> to provide support for images as well. So if anyone can advice if > > HBase > > > >>> is > > > >>> the right solution for me, in conjuction with an imaging tool. > > > >>> > > > >>> Since my team is predominantly Windows based, I would like to know > is > > > it > > > >>> possible to run HBase on a windows machine in stand alone and in > > > >>> clustered > > > >>> mode. > > > >>> > > > >>> Thanks for all your help. > > > >>> > > > >>> > > > >>> nitin > > > >>> > > > >>> > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
