Re: ETL like merge databases to HBase

Shengjie, This is a typical problem statement for data integration. You need to create centralize repository of data coming from different data sources. This centralized data repository (warehouse) will have data refreshed incrementally. This incremental refresh will assure you up-to-date data

Re: ETL like merge databases to HBase

-Actually, it might be easier to go with a pure RDBMS solution here since nowadays the Slave/master architectures in postgre and MySQL are mature enough to handle this sort of thing even for hundreds of thousands of rows. Let's assume RDBMS are from Customer's applications, I don't have that muc

Re: How many column families in one table ?

Regarding slow scan- only fetch the columns /qualifiers that you need. It may be that you are fetching a whole lot of data that you don't need. Try scan.addColumn() and let us know. - R On Sunday, August 4, 2013, lars hofhansl wrote: > BigTable has one more level of abstraction: Locality Groups

Re: How many column families in one table ?

BigTable has one more level of abstraction: Locality Groups A Column Family in HBase is both a Column Faimily and a Locality Group: It is a group of columns *and* it defines storage parameters (compression, versions, TTL, etc). As to how many make sense. It depends. If you can group your columns

Re: How many column families in one table ?

Thanks kevin inder "you are the average of 5 people you spend the most time with" On Aug 4, 2013 8:35 PM, "Kevin O'dell" wrote: > Hi Inder, > > Here is an excellent blog post which is a little dated: > > http://www.larsgeorge.com/2009/11/hbase-vs-bigtable-comparison.html?m=1 > On Aug 4, 2013 1

Re: How many column families in one table ?

Hi Inder, Here is an excellent blog post which is a little dated: http://www.larsgeorge.com/2009/11/hbase-vs-bigtable-comparison.html?m=1 On Aug 4, 2013 10:55 AM, "Inder Pall" wrote: > Kevin > > Would love to hear your thoughts around hbase not big table. > > Thanks > > inder > "you are the

Re: How many column families in one table ?

Kevin Would love to hear your thoughts around hbase not big table. Thanks inder "you are the average of 5 people you spend the most time with" On Aug 4, 2013 8:15 PM, "Kevin O'dell" wrote: > Hi Vimal, > > It really depends on your usage pattern but HBase != Bigtable. > On Aug 4, 2013 2:29 A

Re: How many column families in one table ?

Hi Vimal, It really depends on your usage pattern but HBase != Bigtable. On Aug 4, 2013 2:29 AM, "Vimal Jain" wrote: > Hi, > I have tested read performance after reducing number of column families > from 14 to 3 and yes there is improvement. > Meanwhile i was going through the paper published

Re: Memory distribution for Hadoop/Hbase processes

My questions are : 1) How this thing is working ? It is working because java can over allocate memory. You will know you are using too much memory when the kernel starts killing processes. 2) I just have one table whose size at present is about 10-15 GB , so what should be ideal memory distribution

Re: Memory distribution for Hadoop/Hbase processes

What OS are you using ? What is the output from the following command ? ps aux | grep pid where pid is the process Id for Namenode, Datanode, etc. Cheers On Sun, Aug 4, 2013 at 6:33 AM, Vimal Jain wrote: > Hi, > I have configured Hbase in pseudo distributed mode with HDFS as underlying > stor

Memory distribution for Hadoop/Hbase processes

Hi, I have configured Hbase in pseudo distributed mode with HDFS as underlying storage.I am not using map reduce framework as of now I have 4GB RAM. Currently i have following distribution of memory Data Node,Name Node,Secondary Name Node each :1000MB(default HADOOP_HEAPSIZE property) Hmaster - 5