Re: random read/write performance

There is a performance evaluation result: http://cloudepr.blogspot.com/2009/08/hbase-0200-performance-evaluation.html That benchmarks does not use LZO, we will do it. On Sat, Oct 10, 2009 at 11:21 AM, stack wrote: > I should have said, to figure the count of regions, see the UI (HBase puts > up U

A question of HFile.

Hi stack, In the HFile.java, I found following comments by you. "The current implementation does not offer true multi-threading for reading. The implementation uses FSDataInputStream seek()+read(), which is shown to be much faster than positioned-read call in single thread mode. However, it also

Re: Are you using the Region Historian? Read this

Yes, I also agree to distable historian. The historian should be mined and replayed from logs. On Fri, Sep 18, 2009 at 2:04 PM, stack wrote: > Its a sweet feature, I know how it works, but I find myself never really > using it. Instead I go to logs because there I can get a more > comprehensiv

Re: HFile introduction and it's evaluation

ased on the key. > > Thanks! > -ryan > > On Fri, Sep 11, 2009 at 2:28 PM, Schubert Zhang wrote: > > I like HFile so much, so I analysed/test it and wrote a post to introduce > > it. > > Please access: http://cloudepr.blogspot.com > > May be I have wrong, please comments me. > > >

HFile introduction and it's evaluation

I like HFile so much, so I analysed/test it and wrote a post to introduce it. Please access: http://cloudepr.blogspot.com May be I have wrong, please comments me.

Re: Client API questions

is a significant stability and > security risk. Instead of the above as described there is HBASE-1002 for > Scan and Get. Perhaps it could be extended for Put -- guard predicates which > must evaluate to true for the Put to be applied. > >- Andy > > > > > __

Re: Client API questions

@JG Aside above tow features, I am considering following framework: Sometimes, we want to modify/update a cell value, but the update depends on the current value. IIRC, it is very slow when read+update+write (GET+UPDATE+WRITE). De you think this requirement is reasonable? Maybe we can provide a

Re: Time-series data problem (WAS -> Cassandra vs HBase)

hannels that you mention in another > mail should go in fine... If by chance they have edits that don't fit the > tables current time range, they'll fail to go in because first and last > regions are read-only. > > You should run with bigger regions as the lads suggest elsewhere (a

Re: Cassandra vs HBase

are acceptable. > + How many column families? > one is ok if using HBase. but I am selecting best solution now, > > St.Ack > > > On Wed, Sep 2, 2009 at 11:37 PM, Schubert Zhang wrote: > > > in line. > > > > On Thu, Sep 3, 2009 at 12:46 PM, stack wrote: &

Re: Cassandra vs HBase

s, it is really diffcult and ambitious. So, the time-series data in our case is too large even than any popular websites. > > JG > > On Thu, September 3, 2009 12:15 am, stack wrote: > > On Wed, Sep 2, 2009 at 11:37 PM, Schubert Zhang > > wrote: > > > > &

Re: Cassandra vs HBase

@stack, inline On Thu, Sep 3, 2009 at 3:15 PM, stack wrote: > > When does data expire? Or, how many Billions of rows should your cluster > of > 10-20 nodes carry at a time? about 3 months. hundreds of billinons of rows. > > Will the many rows of same timestamp all go into the one timestamp

Re: Cassandra vs HBase

potential load we are looking at inputing. > > >> > > >> In fact, it's a common treat of such systems, they've been designed > with > > a > > >> certain use case in mind and sometimes I feel like their design and > > >> implementation leak way

Re: Cassandra vs HBase

the >> path of a virtual lock-in. >> >> Now I am not accusing anyone here, just observing that I find it really >> hard to locate any industrial story of those systems in a similar use case >> we have at hand. >> >> The number of nodes this or that c

Re: Cassandra vs HBase

Regardless Cassandra, I want to discuss some questions about HBase/Bigtable. Any advices are expected. Regards runing MapReduce to scan/analyze big data in HBase. Compared to sequentially reading data from HDFS files directly, scan/sequential-reading data from HBase is slower. (As my test, at le

HFile Code Review

Hi stack, I am reviewing the code of HFile. And I will post my comments here. Comments: 1. a minor mistake in HFile.Writer.writeFileInfo(FSDataOutputStream o) int avgValueLen = this.entryCount == 0? 0: (int)(this.keylength/this.entryCount); Here this.keylength should be this.v

Re: Settings

> > HBase: > - > - fs.default.name => hdfs://:9000/ > > This is usually in core-site.xml in Hadoop. Is the client or server needing > this key at all? Did I copy it in the hbase site file by mistake? > [schubert] I think it's better not to copy it into HBase conf file. I suggest you modi

Re: HBase mention in VLDB keynote

empt at some getting some base numbers. > > You really need to load up some of your own data and see how it behaves on > your own cluster. And tuning is increasingly important now as we are > limited by Java GC quite a bit. > > JG > > > Schubert Zhang wrote: > &g

Re: HBase mention in VLDB keynote

om HBase. > > That sounds REALLY interesting! > > - Andy > > > > > > From: Schubert Zhang > To: hbase-user@hadoop.apache.org > Sent: Tuesday, August 25, 2009 8:26:50 PM > Subject: Re: HBase mention in VLDB keynote > >

Re: HBase mention in VLDB keynote

hi andy, Even though current HBase is not yet ready for production, but we know it is really testable and evaluation-able for its data model and architecture. Regards "...and JG's/Ryan's performance test results for 0.20 stand as a contradiction." Can you provide more references? such as a url/li

Re: ANN: hbase 0.20.0 Release Candidate 2 available for download

Hi stack, Yes, you are right, 0.20.0 shoud be released asap to let more people to use and test it. +1 We will continue to test it. Schubert On Tue, Aug 25, 2009 at 12:40 PM, Schubert Zhang wrote: > The .META. table is not flushed. > > But then, I killed -9 all RS and master, then

Re: ANN: hbase 0.20.0 Release Candidate 2 available for download

The .META. table is not flushed. But then, I killed -9 all RS and master, then re-start hbase. Everything become ok. On Tue, Aug 25, 2009 at 12:35 PM, Ryan Rawson wrote: > Did you flush .META. table as well? > > -ryan > > On Mon, Aug 24, 2009 at 9:26 PM, Schubert Zhang wrote: &

Re: ANN: hbase 0.20.0 Release Candidate 2 available for download

pred.Child.main(Child.java:170) On Wed, Aug 19, 2009 at 3:39 AM, Schubert Zhang wrote: > +1 > > I have done our test, it is fine to release. > > On Wed, Aug 19, 2009 at 1:13 AM, stack wrote: > >> Update your binaries. Take down the cluster en masse and update all or do

Re: HBase-0.20.0 Performance Evaluation

The link corrupt, please use this one. http://issues.apache.org/jira/browse/HBASE-1778<http://issues.apache.org/jira/browse/HBASE-1778> On Mon, Aug 24, 2009 at 9:50 PM, Schubert Zhang wrote: > The patch of performance evaluation for 0.20.0 is available @ > http://issues.apache.org

Re: HBase-0.20.0 Performance Evaluation

as > one option. You could buy different machine configurations, one with > large disk, one with less. Or go with modern 8core, 16gb ram machines. > > good luck, > -ryan > > On Tue, Aug 18, 2009 at 2:35 PM, Schubert Zhang wrote: > > @JG and @stack > > > > Helpfu

Re: Tip when migrating your data loading MR jobs from 0.19 to 0.20

So (on default of 3) > you would have to lose 3 nodes at the same instant, which is the > "impossible" paradigm we are working with by using a rep factor of 3. > > JG > > > Schubert Zhang wrote: > >> Thank you J-D, it's a good post. >> I have te

Re: Tip when migrating your data loading MR jobs from 0.19 to 0.20

Thank you J-D, it's a good post. I have test the performance of put.setWriteToWAL(false), it is really fast. And after the batch loading, we shoud call admin.flush(tablename) to flush all data to HDFS. More question about aHDFS support append, and the logs can work as append mode. I think HDFS wil

Re: Exception in rowcount program

This is my setting: hadoop-env.sh export HADOOP_CLASSPATH=${HADOOP_HOME}/../hbase-0.20.0/hbase-0.20.0.jar:${HADOOP_HOME}/../hbase-0.20.0/conf:${HADOOP_HOME}/../hbase-0.20.0/lib/zookeeper-r785019-hbase-1329.jar hbase-env.sh export HBASE_CLASSPATH=${HBASE_HOME}/../hadoop-0.20.0/conf On Fri, Au

Re: data loss with hbase 0.19.3

On Fri, Aug 14, 2009 at 10:03 AM, Chen Xinli wrote: > Thanks Daniel. As you said the latest version has done much to avoid data > loss, would you pls give some example? > > I read the conf file and api, and find some functions related: > 1. in hbase-default.xml, "hbase.regionserver.optionallogflu

Re: ANN: hbase 0.20.0 Release Candidate 2 available for download

+1 I have done our test, it is fine to release. On Wed, Aug 19, 2009 at 1:13 AM, stack wrote: > Update your binaries. Take down the cluster en masse and update all or do > it via a rolling restart: > http://wiki.apache.org/hadoop/Hbase/RollingRestart > . > > Regards whats been fixed since RC1,

Re: HBase-0.20.0 Performance Evaluation

@JG and @stack Helpful! runing RS with 2GB is because we have a heterogeneous node(the slave-5), which has only 4GB RAM. Now, I temporarily removed this node from the cluster. Then we got the ~2ms random-read now. It is fine now. Thank you very much. On Wed, Aug 19, 2009 at 2:52 AM, Jonathan Gra

Re: HBase in a real world application

stack, please assign *HBASE-1778*<https://issues.apache.org/jira/browse/HBASE-1778> to me. On Wed, Aug 19, 2009 at 3:16 AM, Schubert Zhang wrote: > ok, stack. I will done it as soon as possiable. > > > On Wed, Aug 19, 2009 at 2:47 AM, stack wrote: > >> Can you make

Re: HBase in a real world application

ok, stack. I will done it as soon as possiable. On Wed, Aug 19, 2009 at 2:47 AM, stack wrote: > Can you make an issue and a patch please Schubert? > St.Ack > > On Tue, Aug 18, 2009 at 10:52 AM, Schubert Zhang > wrote: > > > We found that there are two issues about t

Re: HBase-0.20.0 Performance Evaluation

ory, but if given sufficient resources, is > much > > better for overall performance/throughput. > > > > Also, do you have Ganglia setup? Are you seeing swapping on your RS > nodes? > > Is there high IO-wait CPU usage? > > > > JG > > > > > &g

Re: HBase in a real world application

We found that there are two issues about the PerformanceEvaluation class. - Is not match for hadoop-0.20.0. - The approach to split map is not strict. Need to provide correct InputSplit and InputFormat classes. And we have just modified the org.apache.hadoop.hbase.PerformanceEvaluation for our eva

Re: Indexed Table in Hbase

The tow approachs of Gary.H and Travis.H are good to work. But I think there is a risk for Travis.H's (columns) approach, when there are many keys for a column value. Then the total size of a index table row may large than a region-size. I think this is not a general approach, you should be very c

Re: HBase-0.20.0 Performance Evaluation

Addition. Only random-reads become very slow, scans and sequential-reads are ok. On Tue, Aug 18, 2009 at 6:02 PM, Schubert Zhang wrote: > stack and J-G, Thank you very much for your helpful comment. > > But now, we find such a critical issue for random reads. > I use sequentic

Re: HBase-0.20.0 Performance Evaluation

status of one region server: request=0.0, regions=13, stores=13, storefiles=14, storefileIndexSize=2, memstoreSize=0, usedHeap=1126, maxHeap=1991, blockCacheSize=338001080, blockCacheFree=79686056, blockCacheCount=5014, blockCacheHitRatio=55 Schubert On Tue, Aug 18, 2009 at 5:02 AM, Schubert Zhang

HBase-0.20.0 Performance Evaluation

We have just done a Performance Evaluation on HBase-0.20.0. Refers to: http://docloud.blogspot.com/2009/08/hbase-0200-performance-evaluation.html

Re: HBase commit autoflush

oh! yes, it is sorted. Thank you very much. On Thu, Aug 13, 2009 at 12:47 PM, Erik Holstad wrote: > Hey Schubert! > The writeBuffer is sorted in processBatchOfRows just like you suggested. > > Regards Erik >

Re: HBase commit autoflush

size of your write buffer. > > JG > > > Schubert Zhang wrote: > >> When set autoflush=false, the client-side write buffer is fine to work. >> But we may not get better performance when random inserting. >> Since the >> >> org.apache.hadoop.hba

HBase commit autoflush

When set autoflush=false, the client-side write buffer is fine to work. But we may not get better performance when random inserting. Since the org.apache.hadoop.hbase.client.HConnectionManager.TableServers.processBatchOfRows() will still send the random rows to different server respectively. In t

Re: Question about HBase

Thanks J-G and Ryan, we are trying to use 0.20.0 now. It is time-consuming, since there is no document now, but we can continue the work. :-) And another general question: Do you think it is possible to store and serve 200TB of data (uncompressed, maybe 50TB after compressed) in a 20-nodes cluste

A question about MapReduce job extracts recent data from HBase/Bigtable

Hi all, I have a periodically scheduled MapReduce job need to extract recent data from a HBase table for analysis, and avoid scanning/reading the analyzed data. Do you have any idea? In the Google paper Section: 8.1 Google Analytics The raw click table (˜200 TB) maintains a row for each end-use

Re: HDFS unbalance issue. (HBase over HDFS)

Sorry "But I found the namenode is fair to process the invalidating for each datanode." should be: "But I found the namenode is unfair to process the invalidating for each datanode." On Fri, Mar 27, 2009 at 3:49 PM, schubert zhang wrote: > Thanks Samuel, > Your i

Re: HDFS unbalance issue. (HBase over HDFS)

ockScanner* thread running on DN will scan, find and > delete these blocks after DN receives the *DNA_INVALIDATE* BlockCommand. > > You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java files, > and find the logic of the garbage collection. Hope it will be helpful. > > On

Re: Data lost during intensive writes

n files each -- very large mapreduce inputs, > intermediates, and outputs -- not that 100s of nodes might > have 1,000s of files open each. In any case, the issue is well > known. > > I have found "dfs.datanode.socket.write.timeout=0" is not > necessary for HBase 0.19

Re: NotServingRegionException as indication of cluster overloading

This NotServingRegionException would happen when assignment.When region-splitting, master should assign the new regions to servers, the processing need time. In this duration, the regions are not accessable. On Thu, Mar 26, 2009 at 7:20 PM, Michael Dagaev wrote: > Hi, all > >It looks like No

Re: Data lost during intensive writes

ncrease the maximum number of xceivers allowed > on each of your datanodes. > > Best regards, > > - Andy > > > From: schubert zhang > > Subject: Re: Data lost during intensive writes > > To: hbase-user@hadoop.apache.org > > Date: Wednesday, March 25, 200

Re: HDFS unbalance issue. (HBase over HDFS)

, Mar 26, 2009 at 7:55 AM, Andrew Purtell wrote: > > > From: schubert zhang > > From another point of view, I think HBase cannot control to > > delete blocks on which node, it would just delete files, and > > HDFS delete blocks where the blocks locating. > > Yes,

Re: Data lost during intensive writes

n Wed, Mar 25, 2009 at 9:36 PM, stack wrote: > On Wed, Mar 25, 2009 at 2:01 AM, schubert zhang wrote: > > > > But the two > > exceptions start to happen earlyer. > > > > Which two exceptions Schubert? > > > hadoop-0.19 > > hbase-0.19.1 (with patch

Re: Question on write optimization

Hi Ryan, Yes, the commit buffer is very useful. Regards "I can import 880m rows in about 90 minutes on a 19 machine cluster.", could you please tell me how many column families and qualifiers in each row? Thank you in advance. Schubert On Wed, Mar 25, 2009 at 5:29 PM, Michael Dagaev wrote: > Th

Re: HDFS unbalance issue. (HBase over HDFS)

the reason of these phenomenons. Thanks. Schubert On Wed, Mar 25, 2009 at 6:37 PM, schubert zhang wrote: > From another point of view, I think HBase cannot control to delete blocks > on which node, it would just delete files, and HDFS delete blocks where the > blocks locating. > > Schube

Re: Data lost during intensive writes

I find if set "dfs.datanode.socket.write.timeout=0", hadoop will always create new socket, is it ok? On Wed, Mar 25, 2009 at 5:01 PM, schubert zhang wrote: > Hi all, > I also meet such same problems/exceptions. > I also have 5+1 machine,e and the system has been running for

Re: HDFS unbalance issue. (HBase over HDFS)

>From another point of view, I think HBase cannot control to delete blocks on which node, it would just delete files, and HDFS delete blocks where the blocks locating. Schubert On Wed, Mar 25, 2009 at 6:28 PM, schubert zhang wrote: > Thanks Ryan. Balancer may take a long time. > >

Re: HDFS unbalance issue. (HBase over HDFS)

ock" hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-23 465927 [schub...@nd1-rack0-cloud logs]$ grep -c "Deleting block" hadoop-schubert-datanode-nd1-rack0-cloud.log 0 On Wed, Mar 25, 2009 at 5:14 PM, Ryan Rawson wrote: > Try > hadoop/bin/start-balancer.sh > > HDFS d

HDFS unbalance issue. (HBase over HDFS)

Hi all, I am using hbase-0.19.1 and hadoop-0.19. My cluster have 5+1 nodes, and there are about 512 regions in HBase (256MB per region). But I found the blocks in HDFS is very unbalanced. Following is the status from HDFS web GUI. (Node: I don't know if this mailing list can display html!) HDFS

Re: Data lost during intensive writes

Hi all, I also meet such same problems/exceptions. I also have 5+1 machine,e and the system has been running for about 4 days, and there are 512 regions now. But the two exceptions start to happen earlyer. hadoop-0.19 hbase-0.19.1 (with patch https://issues.apache.org/jira/browse/HBASE-1008).

Re: RegionServer failure and recovery take a long time

(unless you didn't change that value to false). > > J-D > > On Sat, Mar 21, 2009 at 8:50 PM, schubert zhang wrote: > > Jean Daniel, > > I have done following: > > (1) and -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode, and study the > GC > > detaile

Re: RegionServer failure and recovery take a long time

at org.apache.hadoop.ipc.RPC$Server.call(Unknown Source) at org.apache.hadoop.ipc.Server$Handler.run(Unknown Source) at org.apache.hadoop.ipc.Client.call(Unknown Source) 2042572 ms is really too long. Schubert On Sun, Mar 22, 2009 at 3:01 AM, schubert zhang wrote: > Yes

Re: RegionServer failure and recovery take a long time

Hi Jean-Daniel, It is delightful. Now, my 6-nodes cluster has been running for 12.5 hours with heavy-write, without obvious problem. I have applied your patch. And also add the gc options to hadoop. Thank you for your help. Schubert On Sun, Mar 22, 2009 at 8:50 AM, schubert zhang wrote: > J

Re: RegionServer failure and recovery take a long time

Yes, I missed " ". Thank you. On Sun, Mar 22, 2009 at 2:17 AM, Jean-Daniel Cryans wrote: > Put the options between " " > > J-D > > On Sat, Mar 21, 2009 at 2:15 PM, schubert zhang wrote: > > It's strange when I add -XX:+UseConcMarkSweepGC > &

Re: RegionServer failure and recovery take a long time

even safer. > > J-D > > On Sat, Mar 21, 2009 at 1:34 PM, schubert zhang wrote: > > Jean Daniel, > > Thanks for you kindness. > > Yes, I want more machines, and we will get them soon. :-) > > My application is write-heavy very much. Since my cluster is really > smal

Re: RegionServer failure and recovery take a long time

10. Or instead you might want > to slow down the inserts... It's good to be realist regards what > stress you put on the cluster VS the actual resources. > > J-D > > On Sat, Mar 21, 2009 at 12:24 PM, schubert zhang > wrote: > > Hi Jean Daniel, > > Your help

Re: RegionServer failure and recovery take a long time

chnologies/hotspot/gc/gc_tuning_6.html > for more information. Set this in the hbase-env.sh file on the line > export HBASE_OPTS=... > > - Finally, to make sure that the garbage collection is fast, check if > there is swap. If so, set lower heaps for the MR child processes in

Fwd: RegionServer failure and recovery take a long time

Hi Jean Daniel, I want your help for this issue. I attach the log files, please help analyse it. Thanks. Test env. 5+1 nodes cluster. table: create 'TESTA', {NAME => 'info', VERSIONS => 1, COMPRESSION => 'BLOCK', BLOCKCACHE => true} All test data is generated randomly by a program. HRegionSe

Re: ANN: hbase-0.19.1 release

HI Stack, Is this release same as http://people.apache.org/~stack/hbase-0.19.1-candidate-2/ ? Schubert On Fri, Mar 20, 2009 at 7:50 AM, stack wrote: > HBase 0.19.1 is now available for download from > > http://hadoop.apache.org/hbase

Re: Performance become slower and slower during inserting

, schubert zhang wrote: > Thanks ErikDo you mean I should try to remove the history log files of > mapreduce job in HDFS? In fact, I had disabled it by seting: > hadoop.job.history.user.location > none > > So, there is no big logs in HDFin my cluster. I think the Map

Re: Performance become slower and slower during inserting

Thanks ErikDo you mean I should try to remove the history log files of mapreduce job in HDFS? In fact, I had disabled it by seting: hadoop.job.history.user.location none So, there is no big logs in HDFin my cluster. I think the MapReduce framework is fine. J-D and Stack, I am thinking

Re: Performance become slower and slower during inserting

t your HBase clients > must build up their META cache, which requires lookups in the META > table. > > J-D > > On Thu, Mar 19, 2009 at 1:54 AM, schubert zhang wrote: > > I am testing the performance of HBase, after about one weeks's test. > > I found the HBase beco

Re: NotReplicatedYetException, LeaseExpiredException and one RegionServer Down

: > I've seen HDFS being overwhelmed on very small clusters, like yours, > the way you describe it. The region server probably shut down to not > make things worse with the data. > > J-D > > On Thu, Mar 19, 2009 at 6:43 AM, schubert zhang wrote: > > Report two issue

Re: NotReplicatedYetException, LeaseExpiredException and one RegionServer Down

18:47:52,208 INFO org.apache.hadoop.hbase.master.RegionManager: Skipped 10 region(s) that are in transition states and the HBase is unaccasbale. On Thu, Mar 19, 2009 at 6:43 PM, schubert zhang wrote: > Report two issues I have met: > > Testbed: 1 master + 3 slaves. > > 1. after long time runnin

NotReplicatedYetException, LeaseExpiredException and one RegionServer Down

Report two issues I have met: Testbed: 1 master + 3 slaves. 1. after long time running (batch insert data by mapreduce), sometimes I saw following WARN. Is it caused by network issue or other? 2009-03-19 13:33:43,630 INFO org.apache.hadoop.hbase.regionserver.HLog: removing old log file /hbas

Performance become slower and slower during inserting

I am testing the performance of HBase, after about one weeks's test. I found the HBase become more and more slow when inserting data. (3 regionserver, HBase 0.19.1 and hadoop 0.19.2) Each row have about 32 column (in one family), the row have about 400 bytes raw data. For example: 1. when there

Re: Tow issues for sharing, when using MapReduce to insert rows to HBase

at org.apache.hadoop.mapred.Child.main(Child.java:158) On Wed, Mar 18, 2009 at 3:38 PM, schubert zhang wrote: > Jean Daniel, > Now, I have running HBase0.19.1 candidate 2 and Hadoop 0.19.2 (The code > is got from hadoop branch-0.19, says the MapReduce bug has been fixed in > this branch.) &

Re: Tow issues for sharing, when using MapReduce to insert rows to HBase

Jean Daniel, Now, I have running HBase0.19.1 candidate 2 and Hadoop 0.19.2 (The code is got from hadoop branch-0.19, says the MapReduce bug has been fixed in this branch.) I am verifying, and need some hours for examination. Schubert On Wed, Mar 18, 2009 at 10:14 AM, schubert zhang wrote

Re: Tow issues for sharing, when using MapReduce to insert rows to HBase

t; > J-D > > On Tue, Mar 17, 2009 at 9:33 PM, schubert zhang wrote: > > Hi Jean Daniel, > > I want to try the HBase0.19.1. > > And since the mapreduce of hadoop-0.19.1 is buggy, I want to use a stable > > hadoop 0.18.3, is the HBase 0.19.x can work on hadoop 0.1

Re: Tow issues for sharing, when using MapReduce to insert rows to HBase

Hi Jean Daniel, I want to try the HBase0.19.1. And since the mapreduce of hadoop-0.19.1 is buggy, I want to use a stable hadoop 0.18.3, is the HBase 0.19.x can work on hadoop 0.18.3? Schubert On Wed, Mar 18, 2009 at 12:26 AM, schubert zhang wrote: > Jean Daniel, > Thank you very much

Re: Tow issues for sharing, when using MapReduce to insert rows to HBase

gt; http://www.sollers.ca/blog/2008/swappiness/ > > J-D > > On Tue, Mar 17, 2009 at 3:49 AM, schubert zhang wrote: > > This the "top" info of a regionserver/datanode/tasktracker node.We can > see > > the HRegionServer node is very heavy-loaded. > > >

Re: Tow issues for sharing, when using MapReduce to insert rows to HBase

848 S 218 20.2 2:55.21 java (HRegionServer) 20545 schubert 24 0 3461m 98m 9356 S 15 2.5 10:49.64 java (DataNode) 22677 schubert 24 0 1417m 107m 9224 S9 2.7 0:08.86 java (MapReduce Child) On Tue, Mar 17, 2009 at 3:38 PM, schubert zhang wrote: > Hi all, > > I am runn

Tow issues for sharing, when using MapReduce to insert rows to HBase

Hi all, I am running a MapReduce Job to read files and insert rows into a HBase table. It like a ETL procedure in database world. Hadoop 0.19.1, HBase 0.19.0, 5 slaves and 1 master, DELL2950 server with 4GB memory and 1TB disk on each node. Issue 1. Each time the MapReduce Job eat some files.

Re: Metadata and region mismatch

I find the "ulimit nofile" of one node of my cluster is not enlarged. May my issue is cause by it. I will retest. Thank you very much. and thank J-D very much. Refer to: item 6 of http://wiki.apache.org/hadoop/Hbase/FAQ On Fri, Mar 13, 2009 at 6:09 PM, schubert zhang wrote: >

Re: RangeServer Shutdown itself when DFSClient Exception

many open files > > Did you change the nofile in /etc/security/limits.conf? > > J-D > > On Fri, Mar 13, 2009 at 3:47 PM, schubert zhang wrote: > > Sorry, I am later to check the logs of HDFS. It seems it is caused by > > HDFS DataXceiver! > > > > Near thi

Re: HRegionServer: endless FileNotFoundException in log file

aused by" > clause following it? Hard to tell anything from these excerpts. Can > you grep your logs to see any occurrence of "Too many open files" too? > > Finally, if you could drop by on the IRC channel it would be great, we > will help you digging in the logs. > >

Re: RangeServer Shutdown itself when DFSClient Exception

he datanodes logs? > > Thx, > > J-D > > On Fri, Mar 13, 2009 at 9:23 AM, schubert zhang wrote: > > HBase 0.19.0. > > 5 rangeservers and 1 master, overlay hadoop (5+1). > > > > When inserting rows, one range

Re: HRegionServer: endless FileNotFoundException in log file

> I'd like to know more about your hardware. > > Also I strongly suggest that you try out the 0.19.1 RC1, it has really > nice fixes. > > Thx for reporting all that stuff, > > J-D > > On Fri, Mar 13, 2009 at 5:41 AM, schubert zhang wrote: > > I have a 5+

RangeServer Shutdown itself when DFSClient Exception

HBase 0.19.0. 5 rangeservers and 1 master, overlay hadoop (5+1). When inserting rows, one rangeserve Shutdown itself. 2009-03-13 19:11:40,335 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at: 10.24.1.20:60020 2009-03-13 19:11:40,336 ERROR org.apache.hadoop.hdfs.DFSClien

Re: Metadata and region mismatch

m seems to be an issue fixed in the new hbase 0.19.1 >> release >> candidate: See HBASE-1121 'Cluster confused about where -ROOT- is'. >> >> Worrying is that even after a restart, you cannot get to the troublesome >> region. Is it deployed on a regionserver?

HRegionServer: endless FileNotFoundException in log file

I have a 5+1 HBase/Hadoop cluster. (5 region server and 1 master)A table TESTTAB with only one column family with 36 qualifiers. A process in the master node use batchUpdate (autoFlush=false) to insert random rows into this table. After about 70,000,000 rows inserted, failure. And on the web GUI o

Re: Metadata and region mismatch

21 'Cluster confused about where -ROOT- is'. > > Worrying is that even after a restart, you cannot get to the troublesome > region. Is it deployed on a regionserver? If so, anything pertinent in > the > logs regards this region? > > St.Ack > > On Thu, Mar 12, 2009 a

Re: Metadata and region mismatch

5565...@2008-12-01 17:16:55.117 13576301...@2008-12-08 13:57:43.163 but when I try to get get 13575565...@2008-12-01 17:16:55.117, nothing returned. It seems this region is gone. On Thu, Mar 12, 2009 at 7:09 PM, schubert zhang wrote: > Hi all, > Today, I encounter a new issue about fail

Metadata and region mismatch

Hi all, Today, I encounter a new issue about failure to batchUpdate commit. I am running a program to insert rows into a HBase table, but after long time of batchUpdating, following exception occur: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server Some ser

Re: HRegionServer: Scanner, lease expired issue.

your map tasks that requires your > client to take more than 1 minute to do next()? If it is something > that requires more than 1 minute, do consider changing that value in > you hbase-site.xml to something like 12 millis. > > J-D > > On Wed, Mar 11, 2009 at 5:30 AM, s

Re: HRegionServer: Scanner, lease expired issue.

I made more test. I think my guess may be the cause. In a latest test, there are 11 killed map tasks, and there are also 11 expired leases. I think may be the MapReduce code should be improved. Schubert On Wed, Mar 11, 2009 at 4:38 PM, schubert zhang wrote: > Hi all, > I am running a Map

HRegionServer: Scanner, lease expired issue.

Hi all, I am running a MapReduce job to read regions(row ranges) in Map task. The job can work completely and correctly. But, usually, I find following INFO in hbase regionservre's log file. 2009-03-11 15:57:06,428 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -6710483098243538

Re: Many columns in 0.19

Cool, the HFile solution is what mentioned in Paper of Bigtable, it will be more efficient than MapFile.We are looking forward 0.20.0, including Bloom Filter. Thanks. On Wed, Mar 11, 2009 at 2:28 AM, Jonathan Gray wrote: > Aseem, > > Almost! > > You will have 10 HStores as you say. Each of thos

BloomFilter Failed when compaction

HBase 0.19.0. I know HBASE-1039, but it still happen. 2009-03-11 00:05:41,424 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region TESTTABLE,13666093...@2009-02-1202:33:30.135,1236694689781 2009-03-11 00:05:41,466 WARN o

Re: MR Job question

ich rows you want to get. > > Best Regards. > Slava. > > > On Wed, Mar 4, 2009 at 1:31 PM, schubert zhang wrote: > > > In my job, I can tell the MR job the startRow and endRow, i.e. a row > > range. Then my MR job can only scan the region(s) in the range, and >

Re: MR Job question

er some rows and not > all rows. > > Thank You and Best Regards. > Slava. > > > On Wed, Mar 4, 2009 at 8:57 AM, stack wrote: > > > On Tue, Mar 3, 2009 at 6:17 PM, schubert zhang > wrote: > > > > > Yes, we can tell HBase API only scan rows start wit

Re: MR Job question

for MR job, you should write you code to call HBase API. On Wed, Mar 4, 2009 at 1:14 PM, Slava Gorelik wrote: > Yes, but as I understand this is a not MR Job. This is a scanner usage. > Best Regards. > Slava. > > On Wed, Mar 4, 2009 at 4:17 AM, schubert zhang wrote: > &g

Re: MR Job question

hubert On Wed, Mar 4, 2009 at 3:10 AM, Slava Gorelik wrote: > Thank You for the answer.How can you tell to MR jobs which rows you want to > get ? Is it possible to tell to MR Job give me only rows that starts with > some key ? > > Best Regards. > Slava > > On Tue, Mar 3, 2

  1   2   >