Re: Hbase: TransactionManager: Create table

2015-06-12 Thread Ted Yu
: The article starts with Apache HBase (TM)) - does is stand for Transaction Manager? Apache HBase (TM) is not an ACID compliant database ... -Original Message- From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Friday, June 12, 2015 9:20 PM To: user@hbase.apache.org Cc: C

Re: Hbase: TransactionManager: Create table

2015-06-12 Thread Ted Yu
On the ACID semantices page, I didn't find the term 'transaction manager'. Can you clarify your question ? w.r.t. table creation, please see HBASE-12439 'Procedure V2' and its related tasks. Thanks On Thu, Jun 11, 2015 at 11:07 PM, hariharan_sethura...@dell.com wrote: Hi, Would like to

Re: Has anybody used hbase client 1.0.x in karaf environment

2015-06-11 Thread Ted Yu
Looks like the attached image didn't go through. Mind giving the stack trace for the NoClassDefFoundError ? Thanks On Wed, Jun 10, 2015 at 10:10 PM, hariharan_sethura...@dell.com wrote: Hi hbase users, [Reposting this issue] Hbase-client 0.94.x is working fine in karaf environment. We

Re: HConnection thread waiting on blocking queue indefinitely

2015-06-11 Thread Ted Yu
at 11:42 PM, mukund murrali mukundmurra...@gmail.com wrote: Kindly look into this for full trace of RS. http://pastebin.com/VS17vVd8 Thanks On Wed, Jun 10, 2015 at 11:35 AM, Ted Yu yuzhih...@gmail.com wrote: Can you pastebin the complete stack trace for the region server ? Thanks

Re: Table Delete in Apache Hbase

2015-06-11 Thread Ted Yu
Looks like the questions would be better answered on Phoenix mailing list. On Thu, Jun 11, 2015 at 11:47 AM, Arun Kumaran Sabtharishi arun1...@gmail.com wrote: Hello Hbase users and developers, I have a few questions regarding how table table delete works in hbase. *What I know:* If a

Re: HConnection thread waiting on blocking queue indefinitely

2015-06-10 Thread Ted Yu
Can you pastebin the complete stack trace for the region server ? Thanks On Jun 9, 2015, at 10:52 PM, mukund murrali mukundmurra...@gmail.com wrote: We are using HBase-1.0.0. Just before the client stalled, in RS there were few handler threads that were blocked for MVCC(thread stack

Re: TimeStamp Range Filter for scanning in Hbase ?

2015-06-10 Thread Ted Yu
You can utilize the following method of Scan: public Scan setTimeRange(long minStamp, long maxStamp) To apply Filter, use this: public Scan setFilter(Filter filter) { FYI On Wed, Jun 10, 2015 at 10:33 AM, Devpriya Dave devpr...@yahoo-inc.com.invalid wrote: Hello I am an Intern at

Re: Potential bugs in HTable In incrementColumnValue method

2015-06-09 Thread Ted Yu
I see code in this formation in 0.98 branch. Looking at the unit tests which exercise incrementColumnValue(), they all call: public long incrementColumnValue(final byte [] row, final byte [] family, final byte [] qualifier, final long amount) Possibly because the one mentioned by Jerry is

Re: Potential bugs in HTable In incrementColumnValue method

2015-06-09 Thread Ted Yu
Regards, Jerry On Tue, Jun 9, 2015 at 12:03 PM, Ted Yu yuzhih...@gmail.com wrote: I see code in this formation in 0.98 branch. Looking at the unit tests which exercise incrementColumnValue(), they all call: public long incrementColumnValue(final byte [] row, final byte [] family

Re: Potential bugs in HTable In incrementColumnValue method

2015-06-09 Thread Ted Yu
Did a quick search in HBase JIRA - no hit. Jerry: Mind logging one ? Thanks On Tue, Jun 9, 2015 at 3:30 PM, Andrew Purtell apurt...@apache.org wrote: Is there a JIRA for this? On Tue, Jun 9, 2015 at 11:15 AM, Ted Yu yuzhih...@gmail.com wrote: Seems a bug to me w.r.t. interpretation

Re: Hbase Timestamp Queries

2015-06-09 Thread Ted Yu
Is HBaseStorage class in hbase repo ? I can't find it in 0.98 or hbase-1 branches. Cheers On Tue, Jun 9, 2015 at 3:51 AM, Krishna Kalyan krishnakaly...@gmail.com wrote: Yes you can. Have a look at the HBaseStorage class. On 9 Jun 2015 1:04 pm, Talat Uyarer ta...@uyarer.com wrote: Hi,

Re: HBase shell providing wrong results with startrow(with composite key having String and Ints)

2015-06-09 Thread Ted Yu
to match against non-string data in hbase shell. We should always use double quotes? Even for matching values of cells? On Mon, Jun 8, 2015 at 9:23 PM, Ted Yu yuzhih...@gmail.com wrote: Double quotes allow you to do string interpolation. Aother difference (one pertinent to Anil's question

Re: HBase shell providing wrong results with startrow(with composite key having String and Ints)

2015-06-08 Thread Ted Yu
Double quotes allow you to do string interpolation. Aother difference (one pertinent to Anil's question) is that 'escape sequence' does not work using single quote. Cheers On Mon, Jun 8, 2015 at 9:11 PM, anil gupta anilgupt...@gmail.com wrote: Hi Jean, My bad. I gave a wrong illustration.

Re: Iterate hbase resultscanner

2015-06-08 Thread Ted Yu
From your description, the conversion inside for(Result rs:ListofResult) loop was correct. Since Listclass is custom, probably you need to show us how it is implemented. Which hbase release are you using ? On Mon, Jun 8, 2015 at 9:19 AM, beeshma r beeshm...@gmail.com wrote: HI I have weired

Re: Query on OutOfOrderScannerNextException

2015-06-06 Thread Ted Yu
HBASE-13090 'Progress heartbeats for long running scanners' solves the problem you faced. It is in the 1.1.0 release. FYI On Sat, Jun 6, 2015 at 12:54 PM, Arun Mishra arunmis...@me.com wrote: Hello, I have a query on OutOfOrderScannerNextException. I am using hbase 0.98.6 with 45 nodes.

Re: How to do a fast range scan on a prefix

2015-06-05 Thread Ted Yu
? On Fri, Jun 5, 2015 at 2:22 PM, Ted Yu yuzhih...@gmail.com wrote: You can utilize PrefixFilter. See example in http://hbase.apache.org/book.html#scan On Fri, Jun 5, 2015 at 11:18 AM, jeremy p athomewithagroove...@gmail.com wrote: Assume that my keys look like this : bar:0

Re: How to do a fast range scan on a prefix

2015-06-05 Thread Ted Yu
You can utilize PrefixFilter. See example in http://hbase.apache.org/book.html#scan On Fri, Jun 5, 2015 at 11:18 AM, jeremy p athomewithagroove...@gmail.com wrote: Assume that my keys look like this : bar:0 bar:1 bar:2 baz:0 baz:1 foo:0 foo:1 foo:2 How do I do a fast range scan that

Re: How are version conflicts handled in HBase?

2015-06-05 Thread Ted Yu
question? -Vlad On Thu, Jun 4, 2015 at 6:30 PM, Ted Yu yuzhih...@gmail.com wrote: Dia: Please read http://hbase.apache.org/book.html#_store Cheers On Thu, Jun 4, 2015 at 6:02 PM, Vladimir Rodionov vladrodio...@gmail.com wrote: Yes, last write wins (with higher

Re: How are version conflicts handled in HBase?

2015-06-04 Thread Ted Yu
Dia: Please read http://hbase.apache.org/book.html#_store Cheers On Thu, Jun 4, 2015 at 6:02 PM, Vladimir Rodionov vladrodio...@gmail.com wrote: Yes, last write wins (with higher sequenceId). MemStore will resolve this conflict and only the last put will be added eventually, unless ...

Re: Hbase Bulk load - Map Reduce job failing

2015-06-03 Thread Ted Yu
Looks like you're using MapR FS. Have you considered posting this question on their mailing list ? Cheers On Tue, Jun 2, 2015 at 11:14 PM, Shashi Vishwakarma shashi.vish...@gmail.com wrote: Hi I have map reduce job for hbase bulk load. Job is converting data into Hfiles and loading into

Re: No serialized HRegionInfo

2015-06-02 Thread Ted Yu
AM, Ted Yu yuzhih...@gmail.com wrote: The pastebin of master log snippet doesn't provide enough clue. Can you trace, say, 1042331bd51614fabf01034113804665, back in master log to see whether it went through splits ? Please pastebin related portion of master log. Cheers On Fri, May

Re: HConnection thread waiting on blocking queue indefinitely

2015-06-02 Thread Ted Yu
Which release of hbase are you using ? Is the server hosting hbase:meta operating normally ? Thanks On Tue, Jun 2, 2015 at 4:29 AM, Meeran mohamed.mohid...@zohocorp.com wrote: Hi All, HBase client waits indefinitely for performing any operation. On taking, jstack we saw all the threads

Re: zookeeper closing socket connection exception

2015-06-02 Thread Ted Yu
How much heap did you give the region server ? How much total memory does the box have ? I guess you have read http://hbase.apache.org/book.html#jvm If you're using jdk 1.7.0_60 or newer, you can consider using G1GC. Cheers On Tue, Jun 2, 2015 at 3:26 AM, jeevi tesh jeevitesh...@gmail.com

Re: hfile.bucket.BucketAllocatorException: Allocation too big size

2015-06-01 Thread Ted Yu
Which hbase release are you using ? I seem to recall that hbase.bucketcache.bucket.sizes was the key. Cheers On Mon, Jun 1, 2015 at 7:04 AM, Dejan Menges dejan.men...@gmail.com wrote: Hi, I'm getting messages like: 015-06-01 14:02:29,529 WARN

Re: hfile.bucket.BucketAllocatorException: Allocation too big size

2015-06-01 Thread Ted Yu
with bunch of patches (from Hortonworks). Let me try with that key, on my way :) On Mon, Jun 1, 2015 at 4:19 PM Ted Yu yuzhih...@gmail.com wrote: Which hbase release are you using ? I seem to recall that hbase.bucketcache.bucket.sizes was the key. Cheers On Mon, Jun 1, 2015 at 7:04 AM

Re: zookeeper closing socket connection exception

2015-06-01 Thread Ted Yu
How many zookeeper servers do you have ? Cheers On Mon, Jun 1, 2015 at 12:15 PM, jeevi tesh jeevitesh...@gmail.com wrote: Hi, I'm running into this issue several times but still not able resolve kindly help me in this regard. I have written a crawler which will be keep running for several

Re: hbase get row time out

2015-05-31 Thread Ted Yu
Can you give us a bit more information ? the release of hbase you're using the load on the region server(s) which hosted the row you were retrieving the values for configuration: hbase.client.scanner.timeout.period, hbase.rpc.timeout Thanks On Sun, May 31, 2015 at 4:10 AM, Mohamed Ghareb

Re: No serialized HRegionInfo

2015-05-30 Thread Ted Yu
wrote: On Thu, May 28, 2015 at 7:53 PM, Ted Yu yuzhih...@gmail.com wrote: What release of hbase are you using ? I'm using 0.98.9. I should have mentioned it before anything else. If you trace 5fe111d3302cefb2e96d541a489ff026 in master log, do you find some clue ? From the timestamp

Re: Hbase vs Cassandra

2015-05-29 Thread Ted Yu
See http://hbase.apache.org/book.html#perf.network.call_me_maybe Cheers On Fri, May 29, 2015 at 12:20 PM, Lukáš Vlček lukas.vl...@gmail.com wrote: As for the #4 you might be interested in reading https://aphyr.com/posts/294-call-me-maybe-cassandra Not sure if there is comparable article

Re: HBase snapshots and Compaction

2015-05-28 Thread Ted Yu
this? Thank you. Kind regards, Akmal Abbasov On 28 May 2015, at 16:54, Ted Yu yuzhih...@gmail.com wrote: bq. Completed major compaction of 4 file(s) in s of metrics, V\xA36\x56\x5E\xC5}\xA1\x43\x00\x32\x00T\x1BU\xE0, 3547f43afae5ac3f4e8a162d43a892b4.1417707276446

Re: No serialized HRegionInfo

2015-05-28 Thread Ted Yu
What release of hbase are you using ? If you trace 5fe111d3302cefb2e96d541a489ff026 in master log, do you find some clue ? Pastebin related log snippet if possible. Thanks On Thu, May 28, 2015 at 6:48 PM, Tom Chan tc...@prediction.io wrote: Hi, I kept running into this whenever I remove

Re: HBase snapshots and Compaction

2015-05-28 Thread Ted Yu
of minor compaction. I have this messages all over the log file. Thank you! On 12 May 2015, at 23:53, Ted Yu yuzhih...@gmail.com wrote: Can you pastebin major compaction related log snippets ? See the following for example of such logs: 2015-05-09 10:57:58,961 INFO

Re: HBase snapshots and Compaction

2015-05-28 Thread Ted Yu
it is including 10.8M file? Which setting should be tuned to avoid this? Thank you. Kind regards, Akmal Abbasov On 28 May 2015, at 16:54, Ted Yu yuzhih...@gmail.com wrote: bq. Completed major compaction of 4 file(s) in s of metrics, V\xA36\x56\x5E\xC5}\xA1\x43\x00\x32\x00T\x1BU\xE0

Re: How to set Timeout for get/scan operations without impacting others

2015-05-27 Thread Ted Yu
386.7 M 2015-05-18 18:34:34,032 INFO [main] compress.CodecPool: Got brand-new decompressor [.snappy] *Scanned kv count - 13387493* Any thought or suggestion? Also if it is corrupted file, do you have guidance/link showing how to fix that? Thanks, Mike *From:* Ted Yu [mailto:yuzhih

Re: Best way to benchmark HBase performance

2015-05-26 Thread Ted Yu
Dejan: Assuming your new cluster is 0.98+, you need ycsb which can connect to 0.98+ cluster. See past thread: http://search-hadoop.com/m/YGbbQYu18dTYjB Cheers On Tue, May 26, 2015 at 8:23 AM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi Dejan, YCSB is a very good start. You can

Re: HBase can not put data after deleteall operation

2015-05-25 Thread Ted Yu
If that was the case, please take a look at the following JIRA: HBASE-13734 Improper timestamp checking with VisibilityScanDeleteTracker Cheers On Wed, May 20, 2015 at 7:57 PM, Ted Yu yuzhih...@gmail.com wrote: Did org.apache.hadoop.hbase.security.visibility.VisibilityController happen

Re: avoiding hot spot for timestamp prefix key

2015-05-22 Thread Ted Yu
, then will it search in both the regions R1 and R2 ? On Fri, May 22, 2015 at 10:48 AM, Ted Yu yuzhih...@gmail.com wrote: Does guid change with every key ? bq. use second part of key I don't think so. Suppose first row in the parent region is '1432104178817#321'. After split, the first

Re: DNS mismatch between master and regionserver causes doubly registered regionservers

2015-05-22 Thread Ted Yu
Bryan: HBASE-12954 introduced config for region server hostname. The following added config for master hostname: HBASE-13481 Master should respect master (old) DNS/bind related configurations I will link the above JIRA to HBASE-12954 Cheers On Fri, May 22, 2015 at 12:55 PM, Bryan Beaudreault

Re: Coprocessor accessibility

2015-05-22 Thread Ted Yu
shell .or by default no attribute are set through shell's get . -Original Message- From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Thursday, May 21, 2015 9:29 PM To: user@hbase.apache.org Subject: Re: Coprocessor accessibility Is your hbase deployed in a secure cluster ? Here

Re: Many rows vs many columns

2015-05-22 Thread Ted Yu
Have you looked at opentsdb ? http://opentsdb.net/docs/build/html/user_guide/backends/hbase.html What access pattern to your data are you expecting ? Cheers On Fri, May 22, 2015 at 12:50 AM, Dominik Hübner cont...@dhuebner.com wrote: Is it better to aggregate more data in a single row with

Re: DNS mismatch between master and regionserver causes doubly registered regionservers

2015-05-22 Thread Ted Yu
bq. hbase-1.1.0.1 To my knowledge, latest release was 1.1.0. The release before that was 1.0.1 Can you clarify ? Thanks On Fri, May 22, 2015 at 12:23 PM, Bryan Beaudreault bbeaudrea...@hubspot.com wrote: Thank you Esteban. I checked two different versions: - hbase-1.0.0-cdh5.4.0 (this is

Re: Frequent flush and small compactions, region server load is very high

2015-05-21 Thread Ted Yu
Kudo goes to Bryan Beaudreault http://search-hadoop.com/?q=fc_author=Bryan%20Beaudreault. Cheers On Wed, May 20, 2015 at 11:35 PM, Louis Hust louis.h...@gmail.com wrote: I will try the -XX:G1HeapRegionSize option and test again, thanks ted! 2015-05-20 21:29 GMT+08:00 Ted Yu yuzhih

Re: Unable to load Coprocessor jar file in HBase

2015-05-21 Thread Ted Yu
For #1, namehbase.Coprocessor.region.classes/name There was a typo (in the blog as well), the 'C' should be lower case. For #2, please take a look at ./hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java : // add 2 coprocessor by using new

Re: readAtOffset error when reading from HFiles

2015-05-21 Thread Ted Yu
defaulting to the positional read instead of seek+read - is this accurate? On Tue, May 12, 2015 at 12:00 PM, Ted Yu yuzhih...@gmail.com wrote: -w is shorthand for --seekToRow (so it is not offset): -w,--seekToRow arg Seek to this row and print all the kvs

Re: Getting intermittent errors while insertind data into HBase

2015-05-21 Thread Ted Yu
bq. Stack Trace: java.util.LinkedList$ListItr.checkForComodification( LinkedList.java:953) What was the error prior to the above line ? Please let us know the hbase release you're using. Cheers On Thu, May 21, 2015 at 2:08 AM, Serega Sheypak serega.shey...@gmail.com wrote: Maybe you share

Re: avoiding hot spot for timestamp prefix key

2015-05-21 Thread Ted Yu
Does guid change with every key ? bq. use second part of key I don't think so. Suppose first row in the parent region is '1432104178817#321'. After split, the first row in first daughter region would still be '1432104178817#321'. Right ? Cheers On Thu, May 21, 2015 at 9:57 PM, Shushant Arora

Re: Coprocessor accessibility

2015-05-21 Thread Ted Yu
Is your hbase deployed in a secure cluster ? Here is signature for preGetOp: public void preGetOp(final ObserverContextRegionCoprocessorEnvironment e, final Get get, final ListCell results) throws IOException { User who wants this observer activated can pass attribute, with known name,

Re: How to know the root reason to cause RegionServer OOM?

2015-05-20 Thread Ted Yu
no evidence to prove my idea, so hope more helps. Thanks. At 2015-05-20 10:04:19, Ted Yu yuzhih...@gmail.com wrote: For scenario #1, you would see in the regionserver.out file that kill -9 command was applied due to OOME. For scenario #2, can you see if dmesg provides some clue

Re: Frequent flush and small compactions, region server load is very high

2015-05-20 Thread Ted Yu
', MIN_VERSIONS = '0', TTL = 'FOREVER', KEEP_DELETED_CELLS = 'FALSE', BLOCKSIZE = '65536', IN_MEMORY = 'false', BLOCKCACHE = 'true'} 1 row(s) in 1.0260 seconds 2015-05-20 10:59 GMT+08:00 Ted Yu yuzhih...@gmail.com: How many column families does your table have ? Can you show us

Re: HBase can not put data after deleteall operation

2015-05-20 Thread Ted Yu
Did org.apache.hadoop.hbase.security.visibility.VisibilityController happen to be one of the coprocessor(s) loaded ? Cheers On Wed, May 20, 2015 at 7:40 PM, xufeng xufeng...@163.com wrote: hi everyone: I encount a HBase issue: HBase can not put data after deleteall operation My HBase

Re: Frequent flush and small compactions, region server load is very high

2015-05-19 Thread Ted Yu
How many column families does your table have ? Can you show us the output of describe 'ttsi_order:ttsi_order_process' ? Thanks On Tue, May 19, 2015 at 4:56 AM, Louis Hust louis.h...@gmail.com wrote: Hi, all My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region server load

Re: Scan vs Get

2015-05-19 Thread Ted Yu
and you will get the same performances. Scan and get without the QC returns in very similar time. 191ms for one, 194ms for the other one. 2015-05-19 23:02 GMT-04:00 Ted Yu yuzhih...@gmail.com: J-M: How many times did you try the pair of queries ? Since scan was run

Re: Scan vs Get

2015-05-19 Thread Ted Yu
J-M: How many times did you try the pair of queries ? Since scan was run first, this would give the get query some advantage, right ? Cheers On Tue, May 19, 2015 at 7:34 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Are not Scan and Gets supposed to be almost as fast? I have a

Re: Re: Re: How to know the root reason to cause RegionServer OOM?

2015-05-19 Thread Ted Yu
For scenario #1, you would see in the regionserver.out file that kill -9 command was applied due to OOME. For scenario #2, can you see if dmesg provides some clue ? Cheers On Tue, May 19, 2015 at 6:32 PM, David chen c77...@163.com wrote: Thanks for guys reply, its indeed helped me. Another

Re: HMaster restart with error

2015-05-18 Thread Ted Yu
] client.HConnectionManager$HConnectionImplementation: Checking master connection Does this mean the cluster was not operational? 2015-05-18 11:45 GMT+08:00 Ted Yu yuzhih...@gmail.com: After l-namenode1 became active master , it assigned regions: 2015-05-15 12:16:40,432 INFO [master:l-namenode1

Re: Where can I find the document for specific version of hbase api ?

2015-05-18 Thread Ted Yu
Xiaobo: Can you download the source tarball for the release you're using ? You can find all the API information from the source code. Cheers On Mon, May 18, 2015 at 1:33 AM, guxiaobo1982 guxiaobo1...@qq.com wrote: Hi, http://hbase.apache.org/apidocs/ shows the latest version, but where I

Re: How to set Timeout for get/scan operations without impacting others

2015-05-18 Thread Ted Yu
hbase.client.operation.timeout is used by HBaseAdmin operations, by RegionReplicaFlushHandler and by various HTable operations (including Get). hbase.rpc.timeout is for the RPC layer to define how long HBase client applications take for a remote call to time out. It uses pings to check

Re: How to set Timeout for get/scan operations without impacting others

2015-05-18 Thread Ted Yu
) at org.apache.hadoop.hbase.io.hfile.HFileBlock.init(HFileBlock.java:239) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1448 Thanks, Mike *From:* Ted Yu [mailto:yuzhih...@gmail.com] *Sent:* Monday, May 18, 2015 11:55 PM *To:* user

Re: HMaster restart with error

2015-05-17 Thread Ted Yu
$HConnectionImplementation.getKeepAliveMasterService(HConnectionManager.java:1836) Is the exception means the HMaster not working normally or somewhat? 2015-05-17 11:06 GMT+08:00 Ted Yu yuzhih...@gmail.com: bq. the HMaster is handling two region server down, and not ready to handle client request? I didn't mean

Re: HMaster restart with error

2015-05-17 Thread Ted Yu
the cluster was operational this time. Cheers On Sun, May 17, 2015 at 8:24 AM, Ted Yu yuzhih...@gmail.com wrote: bq. the backup master take over at 2015-05-15 12:16:40,024 ? The switch of active master should be earlier than 12:16:40,024 - shortly after 12:15:58 l-namenode1 would do some

Re: Region Server Start Failure

2015-05-17 Thread Ted Yu
Can you give us a bit more information - I assume hbase config is the same on this server compared with the other region servers: region server log preceding 'Failed to close file' exception - I looked for 'Caused by' but didn't find any. is this a secure cluster ? w.r.t.

Re: HMaster restart with error

2015-05-16 Thread Ted Yu
or do something else? 2015-05-16 13:59 GMT+08:00 Louis Hust louis.h...@gmail.com: OK, you can get the log from http://pan.baidu.com/s/1pqS6E 2015-05-16 13:26 GMT+08:00 Ted Yu yuzhih...@gmail.com: Can you check server log on 192.168.39.22 http://l-namenode2.dba.cn8.qunar.com

Re: HMaster restart with error

2015-05-16 Thread Ted Yu
:29 GMT+08:00 Ted Yu yuzhih...@gmail.com: In the period you identified, master was assigning regions. e.g. 2015-05-15 12:13:09,683 INFO [l-namenode2.dba.cn8.qunar.com ,6,1431663090427-GeneralBulkAssigner-0] master.RegionStates: Transitioned {c634280ce287b2d6cebd88b61accf685 state

Re: HMaster restart with error

2015-05-15 Thread Ted Yu
Can you check server log on 192.168.39.22 http://l-namenode2.dba.cn8.qunar.com/192.168.39.22:6 ? That should give you some clue. Cheers On Fri, May 15, 2015 at 8:22 PM, Louis Hust louis.h...@gmail.com wrote: Hi all, I use hbase0.96.0 with hadoop 2.2.0, and the custom said they can not

Re: Re: Re: How to know the root reason to cause RegionServer OOM?

2015-05-15 Thread Ted Yu
I got '502 Bad Gateway' trying to access the post David mentioned. Here is the same article in case you get 502 error: http://java.dzone.com/articles/OOM-relation-to-swappiness FYI On Thu, May 14, 2015 at 2:40 AM, David chen c77...@163.com wrote: Thanks for guys' helps. Maybe the root reason

Re: Major Compacting Multiple Regions in same Region Server

2015-05-15 Thread Ted Yu
Have you tried giving hbase.regionserver.thread.compaction.large value larger than 1 ? Cheers On Fri, May 15, 2015 at 9:11 AM, rahul malviya malviyarahul2...@gmail.com wrote: Hi, I have observered major compaction on our hbase cluster it seems it compacts only one region at a time per

Re: Re: Re: How to know the root reason to cause RegionServer OOM?

2015-05-15 Thread Ted Yu
? At 2015-05-14 01:04:35, Ted Yu yuzhih...@gmail.com wrote: For #2, partial row would be returned. Please take a look at the following method in RSRpcServices around line 2393 : public ScanResponse scan(final RpcController controller, final ScanRequest request) Cheers On Wed, May 13

Re: Upgrading from 0.94 (CDH4) to 1.0 (CDH5)

2015-05-14 Thread Ted Yu
Looks like there is a JIRA already: HBASE-11843 MapReduce classes shouldn't be in hbase-server Cheers On Thu, May 14, 2015 at 10:41 AM, anil gupta anilgupt...@gmail.com wrote: +1 on moving MR related code to hbase-client or we can have a separate artifact called hbase-mapreduce I also have

Re: HBase MapReduce in Kerberized cluster

2015-05-14 Thread Ted Yu
Please take a look at HBASE-12493 User class should provide a way to re-use existing token which went into 0.98.9 FYI On Thu, May 14, 2015 at 8:37 AM, Edward C. Skoviak edward.skov...@gmail.com wrote: I searched (current) 0.98 and branch-1 where I found:

Re: Problem tin scanning hbase in reducer phase of mapreduce

2015-05-14 Thread Ted Yu
Can you give us a bit more information? Such as stack trace if the scan hangs, master / region server log snippet, etc Thanks On May 13, 2015, at 11:04 PM, Priya prt...@gmail.com wrote: I am using hadoop 1.2.1 and hbase 0.94.12 . I have a scenario where in reducer phase i have to read

Re: client Table instance, confused with autoFlush

2015-05-13 Thread Ted Yu
Please take a look at https://issues.apache.org/jira/browse/HBASE-12728 Cheers On Wed, May 13, 2015 at 6:25 AM, Serega Sheypak serega.shey...@gmail.com wrote: Hi, in 0.94 we could use autoFlush method for HTable. Now HTable shouldn't be used, we refactoring code for Table Here is a note:

Re: Problems with Phoenix and HBase

2015-05-13 Thread Ted Yu
Putting dev@ to bcc The question w.r.t. sqlline should be posted to Phoenix user mailing list. w.r.t. question on hbase shell, can you give us more information ? release of hbase you use was there exception from hbase shell when no tables were returned ? master log snippet when above happened

Re: Re: How to know the root reason to cause RegionServer OOM?

2015-05-13 Thread Ted Yu
, will not need HBASE-11544. right? 2. Assume scan has set caching to 1 and maxResultSize to 1, for a row which per cell size is more than 1, will the first RPC return the whole or partial row? I think the whole row, right? At 2015-05-13 11:04:04, Ted Yu yuzhih...@gmail.com wrote

Re: MR against snapshot causes High CPU usage on Datanodes

2015-05-13 Thread Ted Yu
Have you enabled short circuit read ? Cheers On Wed, May 13, 2015 at 9:37 AM, rahul malviya malviyarahul2...@gmail.com wrote: Hi, I have recently started running MR on hbase snapshots but when the MR is running there is pretty high CPU usage on datanodes and I start seeing IO wait message

Re: HBase MapReduce in Kerberized cluster

2015-05-13 Thread Ted Yu
bq. it has been moved to be a part of the hbase-server package I searched (current) 0.98 and branch-1 where I found: ./hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java FYI On Wed, May 13, 2015 at 11:45 AM, Edward C. Skoviak edward.skov...@gmail.com wrote: I'm

Re: HBase snapshots and Compaction

2015-05-12 Thread Ted Yu
Can you show us compaction related parameters you use ? e.g. hbase.hregion.majorcompaction , hbase.hregion.majorcompaction.jitter , etc On Tue, May 12, 2015 at 9:52 AM, Akmal Abbasov akmal.abba...@icloud.com wrote: HI, I am using HBase 0.98.7. I am using HBase snapshots to backup data. I

Re: [VOTE] Third release candidate for HBase 1.1.0 (RC2) is available.

2015-05-12 Thread Ted Yu
+1 - Checked signatures - Ran unit test suite - Exercised basic shell commands - Performed RAT check - Ran LTT in a mixed deployment (0.98.4 servers along side 1.1.0 servers) Cheers On Tue, May 12, 2015 at 3:56 PM, Andrew Purtell apurt...@apache.org wrote: +1 - Checked sums and signatures

Re: HBase snapshots and Compaction

2015-05-12 Thread Ted Yu
12, 2015 at 2:06 PM, Akmal Abbasov akmal.abba...@icloud.com wrote: Hi Ted, Thank you for reply. I am running with the default settings. Sent from my iPhone On 12 May 2015, at 22:02, Ted Yu yuzhih...@gmail.com wrote: Can you show us compaction related parameters you use ? e.g

Re: How to know the root reason to cause RegionServer OOM?

2015-05-12 Thread Ted Yu
Does the following appear in the command which launched region server ? -XX:OnOutOfMemoryError=kill -9 %p There could be multiple reasons for region server process to encounter OOME. Please take a look at HBASE-11544 which fixes a common cause. The fix is in the upcoming 1.1.0 release. Cheers

Re: readAtOffset error when reading from HFiles

2015-05-12 Thread Ted Yu
What release of hbase are you using ? Please read http://hbase.apache.org/book.html#hfile where you can find description about HFile tool. This tool would allow you to investigate given HFile. Cheers On Tue, May 12, 2015 at 6:02 AM, donmai dood...@gmail.com wrote: Hi, I'm getting this error

Re: Regions and Rowkeys

2015-05-12 Thread Ted Yu
Arun: See the following for details: http://hbase.apache.org/book.html#_determining_split_points http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/util/RegionSplitter.HexStringSplit.html Cheers On Tue, May 12, 2015 at 6:11 AM, Talat Uyarer ta...@uyarer.com wrote: Hi Arun, rowKeys.

Re: Mapping Over Cells

2015-05-11 Thread Ted Yu
Please note that if you have a Filter with active filterRowCells(List) or filterRow(), the Filter cannot be set on the Scan when setBatch() is called on the same Scan object. FYI On Mon, May 11, 2015 at 10:12 AM, Mike Axiak m...@axiak.net wrote: Hi Ryan, You can call setBatch() on the scan

Re: [VOTE] First release candidate for HBase 1.1.0 (RC0) is available.

2015-05-11 Thread Ted Yu
+1 on abbreviated voting window. I will give the new RC a spin. On Mon, May 11, 2015 at 11:42 AM, Andrew Purtell apurt...@apache.org wrote: +1 on a new RC FWIW, I have time to check it in an abbreviated voting window. On Mon, May 11, 2015 at 11:40 AM, Nick Dimiduk ndimi...@apache.org

Re: Regions and Rowkeys

2015-05-11 Thread Ted Yu
You can presplit table at time of creation. e.g. table 't1' is split into 15 regions in the following example: hbase(main):002:0 create 't1', 'f1', {NUMREGIONS = 15, SPLITALGO = 'HexStringSplit'} 0 row(s) in 2.3140 seconds The regions of the table would be spread evenly across the region

Re: Not able to Load EndPoint Coprocessor

2015-05-08 Thread Ted Yu
class AvgService is not shown below. Which line is #43 in TagAggregate.java ? Please take a look at the following classes from hbase: hbase-server//src/main/java/org/apache/hadoop/hbase/coprocessor/AggregateImplementation.java

Re: MapReduce on Sanpshots

2015-05-08 Thread Ted Yu
Please see Release Note of HBASE-8369 Cheers On Fri, May 8, 2015 at 8:16 AM, Rahul Malviya malviyarahul2...@gmail.com wrote: Hi All, I have been MapReduce on HBase for a while now and this has been really successful for me. But I have some MRs jobs in my pipeline which scan every row on

Re: Snapshot Timing Out on a Large HBase Table

2015-05-08 Thread Ted Yu
bq. ERROR: Snapshot 'snapshot_test' wasn't completed in expectedTime:6 ms Rahul: Please check your master log around 15:15 (when the timeout happened). You should find some clue there. Cheers On Fri, May 8, 2015 at 11:40 AM, rahul malviya malviyarahul2...@gmail.com wrote: We have 16000

Re: HBase shell noninteractive mode

2015-05-05 Thread Ted Yu
Akmal: When 0.98.13 is released later this month, you can upgrade to 0.98.13 with rolling upgrade. Cheers On Tue, May 5, 2015 at 6:28 AM, Akmal Abbasov akmal.abba...@icloud.com wrote: Hi Sean, Thank you for a quick reply. I am not sure about next upgrade time for now. So the only solution

Re: Right value for hbase.rpc.timeout

2015-05-04 Thread Ted Yu
at 11:27 PM Ted Yu yuzhih...@gmail.com wrote: Please take a look at 98.3 under http://hbase.apache.org/book.html#trouble.client BTW what's the value for hbase.hregion.max.filesize ? Which split policy do you use ? Cheers On Thu, Apr 30, 2015 at 6:59 AM, Dejan Menges dejan.men

Re: RegionServer not crash but not response!!

2015-05-04 Thread Ted Yu
Attachment didn't go through. Please use pastebin. If you can get stack trace for the region server, that would help too. On Sun, May 3, 2015 at 9:35 PM, mail list louis.hust...@gmail.com wrote: Hi, all, Our hbase is hbase-0.96.0, we found one region server not work normally, can any body

Re: HBase Questions

2015-05-03 Thread Ted Yu
For #1, see http://hbase.apache.org/book.html#versions and http://hbase.apache.org/book.html#schema.versions Cheers On Fri, May 1, 2015 at 9:17 PM, Arun Patel arunp.bigd...@gmail.com wrote: 1) Are there any problems having many versions for a column family? What's the recommended limit? 2)

Re: exportSnapshot tool

2015-05-03 Thread Ted Yu
Can you give us a bit more information ? Such as: release of hbase you're using release of hadoop you're using the command line for snapshot export Thanks On Sun, May 3, 2015 at 7:53 AM, Akmal Abbasov akmal.abba...@icloud.com wrote: Hi, I using exportSnapshot tool, and observed a strange

Re: exportSnapshot tool

2015-05-03 Thread Ted Yu
export is: hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snappy -copy-to hdfs://hb1m/hbase -overwrite Thank you Regards, Akmal Abbasov On 03 May 2015, at 16:57, Ted Yu yuzhih...@gmail.com wrote: Can you give us a bit more information ? Such as: release of hbase

Re: HBase copyTable stuck at map 100% reduce 0%

2015-05-02 Thread Ted Yu
Here is usage of CopyTable: System.err.println( $ bin/hbase + org.apache.hadoop.hbase.mapreduce.CopyTable --starttime=1265875194289 --endtime=1265878794289 + --peer.adr=server1,server2,server3:2181:/hbase --families=myOldCf:myNewCf,cf2,cf3 TestTable ); Your peer.adr

Re: jmx?description=true

2015-05-01 Thread Ted Yu
Which release of HBase do you use ? I tried on hbase 1.1.0 RC and saw descriptions when ?description=true is appended to URL. Here is top of region server JMX : { beans : [ { name : java.lang:type=Memory, description : Information on the management interface of the MBean,

Re: [VOTE] First release candidate for HBase 1.1.0 (RC0) is available.

2015-04-30 Thread Ted Yu
Jeetendra: Add the following repo in your pom.xml (repositories section): https://repository.apache.org/content/repositories/orgapachehbase-1076 Then you can use 1.1.0 for hbase version. Cheers On Wed, Apr 29, 2015 at 11:06 PM, Jeetendra Gangele gangele...@gmail.com wrote: I means how to

Re: HBase Filesystem Adapter

2015-04-30 Thread Ted Yu
Please take a look at HBASE-13485 Cheers On Apr 30, 2015, at 6:35 AM, Buğra Çakır bugra.ca...@oranteknoloji.com wrote: Hi, I would like to use HBase with distributed filesystems other than HDFS. Are there any plans for developing filesystem adapters for these distributed

Re: [VOTE] First release candidate for HBase 1.1.0 (RC0) is available.

2015-04-30 Thread Ted Yu
. if this is not possible how to add this repository is Pom.xml? On 30 April 2015 at 19:28, Ted Yu yuzhih...@gmail.com wrote: Jeetendra: Add the following repo in your pom.xml (repositories section): https://repository.apache.org/content/repositories/orgapachehbase-1076

Re: Right value for hbase.rpc.timeout

2015-04-30 Thread Ted Yu
Please take a look at 98.3 under http://hbase.apache.org/book.html#trouble.client BTW what's the value for hbase.hregion.max.filesize ? Which split policy do you use ? Cheers On Thu, Apr 30, 2015 at 6:59 AM, Dejan Menges dejan.men...@gmail.com wrote: Basically how I came to this question -

Re: Rebalancing after manual region move

2015-04-29 Thread Ted Yu
In the upcoming 1.1.0 release, the following would make balancer state inspection more user-friendly: HBASE-13222 Provide means of non-destructive balancer inspection FYI On Wed, Apr 29, 2015 at 4:11 AM, Dejan Menges dejan.men...@gmail.com wrote: Hi Jean-Marc, Ok, that's what I thought too,

<    7   8   9   10   11   12   13   14   15   16   >