Large Files in Column Qualifier

I am evaluating an HBase design that would require that it rarely house a 1GB file in the column qualifier. Files range from 1GB - 1KB. These files are raw files being ingested from clients and to be kept for some period of time (several years) for quality control purposes. The application does not

Re: Large Files in Column Qualifier

That is the other solution I am evaluating. I was attracted by the simplicity and flexibility of keeping these files in HBase rows, but it sounds like the better option is found in using HDFS for file storage and HBase for metadata.

Re: Any guidelines on the number of region servers in HBase cluster?

hbase-site.xml -> hbase.hregion.max.filesize governs the size of the individual region servers. Once it exceed this size it splits. G

Copy Hbase Table Schema

I have to rebuild a test database elsewhere in production and I have about 50 columns in one column family. I would rather not define the schema manually. Could I somehow build the tables in another node without manually typing each column? Thanks, Geo

Re: Copy Hbase Table Schema

; 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ^ ENCODE_ON_DISK => &#x

Re: Copy Hbase Table Schema

TL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'} 0 row(s) in 2.3230 seconds On Sat, Nov 9, 2013 at 12:12 PM, Geovanie Marquez &l

Re: HBase Large Load Issue

What is your distributed hardware/services configuration? Where are your masters and slaves and what spec is maintained by each? You have compaction set to zero but the issues happen near a major compaction event, so are you running manual compactions during a heavy put operation? On Tue, Dec 3,

Re: Why hadoop/hbase uses DNS/hosts/hostname in such a strange way?

This may not answer why it is designed this way, but it should give you more insight into how it is done. Here is how the network resolution

Re: OutOfOrderScannerNextException when export table

Try trimming down hbase.client.scanner.caching to 100 server side and increase hbase.regionserver.lease.period, hbase.client.scanner.timeout.period to 5 minutes and see how it goes. Geo On Jan 15, 2015 6:01 AM, "Li Li" wrote: > I am using hbase-0.98.5-hadoop1 with hadoop 1.2.1. > And I want to e

Re: Streaming data to htable

We use Spark to convert large batches of data directly into HFiles. We've found it to be extremely performant, but we do not batch since our use case is not streaming. We bring it in about 50GB at a time so we would not suffer from the small files issue mentioned, but we do manually manage our regi

Re: Monitoring tools for Hbase

Cloudera Manager if you are using a distributor like them. If not, OpenTSDB together with Grafana allows you to graph anything and annotate it which may be useful when you deploy code that may cause interesting behavior. On Mon, Apr 6, 2015 at 5:33 PM, Siva wrote: > Hi, > > Could someone recomme

Re: regions in transition

check hmaster:60010 under TASKS (between Software Attributes and Tables) you will see if you have regions in transition. This will tell you which regions are transitioning and you can go to those region server logs and check them, I've run into a couple of these and every time they've talk to me ab

Re: regions in transition

RITs On Tue, Dec 22, 2015 at 11:52 AM, Brian Jeltema wrote: > I’m running Ambari 2.0.2 and HPD 2.2. I don’t see any of this displayed at > master:60010. > > I really think this problem is the result of cruft in ZooKeeper. Does > anybody know > if it’s safe to delete the node

Re: RPC Client OutOfMemoryError Java Heap Space

________ > From: Geovanie Marquez [geovanie.marq...@gmail.com] > Sent: Thursday, May 08, 2014 2:35 PM > To: user@hbase.apache.org > Subject: Re: RPC Client OutOfMemoryError Java Heap Space > > sorry didn't include version > > CDH5 version - C

Re: RPC Client OutOfMemoryError Java Heap Space

Is this an expectation problem or a legitimate concern. I have been studying the memory configurations on cloudera manager and I don't seem to see where I can improve my situation. On Thu, May 8, 2014 at 5:35 PM, Geovanie Marquez wrote: > sorry didn't include version > >

Re: RPC Client OutOfMemoryError Java Heap Space

sorry didn't include version CDH5 version - CDH-5.0.0-1.cdh5.0.0.p0.47 On Thu, May 8, 2014 at 5:32 PM, Geovanie Marquez wrote: > Hey group, > > There is one job that scans HBase contents and is really resource > intensive using all resources available to yarn (under Resource

Re: RPC Client OutOfMemoryError Java Heap Space

gt; ? > St.Ack > > > On Tue, May 13, 2014 at 7:07 AM, Geovanie Marquez < > geovanie.marq...@gmail.com> wrote: > > > The following property does exactly what I wanted our environment to do. > I > > had a 4GiB Heap and ran the job and no jobs failed. Then I dropped our

Re: Region servers crashing during mapreduce

It's really not going to be useful to guess without more log investigation.check the master node logs to see when the first region server went down and correlate zookeeper and region server logs to the minute or two before it died. It could be garbage collection or high scan batches killing your s

Re: RPC Client OutOfMemoryError Java Heap Space

example I should follow other than the only mapreduce section already there to update the book? On Thu, May 15, 2014 at 9:46 AM, Geovanie Marquez < geovanie.marq...@gmail.com> wrote: > Thanks for the suggestion - I'll try to get that out this weekend > sometime. > > >

Region Server State Checks

What is a good way of checking RegionState? I am getting the following error when I programmatically try to merge regions. Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hbase.exceptions.MergeRegionException: org.apache.hadoop.hbase.exceptions.MergeRegionException: Unabl

Re: Region Server State Checks

Looks like exactly what I was looking for, thank you! On Tue, Jun 24, 2014 at 2:39 PM, Ted Yu wrote: > Please take a look at the following method in HBaseAdmin: > > public ClusterStatus getClusterStatus() throws IOException { > > Cheers > > > On Tue, Jun 24, 20

unable to merge region - "has merge qualifier"

I have a table that blew up in the number of regions, I fixed the configuration and now need to bring the regions back down to our desired number of 16. I wrote a program to do exactly that and it worked when tested on a single node cluster, but now in production I get perpetually stuck on a merge

Re: unable to merge region - "has merge qualifier"

y is that region 190d50047a820d9b3ef588429c9065ea still has > reference(s) > > From CatalogJanitor#cleanMergeQualifier(): > > if (regionFs == null || !regionFs.hasReferences(htd)) { > > > Cheers > > > On Sat, Jun 28, 2014 at 7:12 AM, Geovanie Marquez < > geovanie.mar

Aggresive compactions

I notice that when I have a regions with store file counts greater than hbase.hstore.blockingStoreFiles, on cluster startup the number drops dramatically under this value to just under the blockingStoreFile parameter value in a relative short amount of time and then it stalls and doesn't fall more

Re: Aggresive compactions

that the server includes all files for all compactions? On Sun, Aug 10, 2014 at 11:46 AM, Ted Yu wrote: > What is the value for the config parameter 'hbase.hstore.compaction.ratio' > ? > > Thanks > > > On Sun, Aug 10, 2014 at 7:17 AM, Geovanie Marquez < > geova

Re: Aggresive compactions

t 11:48 AM, Geovanie Marquez < geovanie.marq...@gmail.com> wrote: > The default: 1.2F > hbase.hstore.compaction.ratio > > For minor compaction, this ratio is used to determine whether a given > StoreFile which is larger than hbase.hstore.compaction.min.size is eligible > for

Non-sensical? Compaction Metrics Hbase UI

What is the meaning of the progress value below? I see the information broken down below in the following HBase UI. http://region-server-ip:60030/rs-status?filter=general#regionCompactStats Region Name Num. ,`\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1398130030457.2dddb2b

Re: Non-sensical? Compaction Metrics Hbase UI

ich release of hbase are you using ? > > Thanks > > > On Mon, Aug 11, 2014 at 9:49 AM, Geovanie Marquez < > geovanie.marq...@gmail.com> wrote: > > > What is the meaning of the progress value below? > > > > > > > > I see the information broken d

Re: Non-sensical? Compaction Metrics Hbase UI

the new hfile size, expecting it to be approximately the same size > as total hfiles pre-compaction. > > > On Mon, Aug 11, 2014 at 3:11 PM, Geovanie Marquez < > geovanie.marq...@gmail.com> wrote: > > > *Attribute Name* *Value* > > HBase Version 0.98.1-cdh5.1.0, rUnknown

Re: Non-sensical? Compaction Metrics Hbase UI

compaction progress is almost never >> accurate. >> >> I've taken to doing: >> >> hdfs -du /hbase///.tmp >> >> to track the new hfile size, expecting it to be approximately the same >> size >> as total hfiles pre-compaction. >> >>

Re: region stuck in failed close state

No it cannot, that region is in transition. Check out hbase hbck for more details, and sometimes hbase hbck repair takes care of this. On May 26, 2016 2:44 AM, "Heng Chen" wrote: > And there is another question about failed close state, does it mean the > region in this state could be read and w

Re: Very High CPU usage in RegionServer

We had a problem like this once. We localized it to a fuzzy row filter request. Are you scanning this region with such a filter? If so, there is a patch out there we applied that got rid of the problem. On Jul 8, 2016 2:58 AM, "Samir Ahmic" wrote: > Hi Sandeep, > > What sort of load is on cluste