Re: OK to mix transactional tables with regular HTables?

Yes, you should be able to mix in the use of transactional tables. You can access tables w/out the transactional semantics/overhead as normal, even when running a TransactionalRegionServer. You also can use the same tables w/out the TRS extension, however edits will not be reflected in indexes. Th

Re: IndexedTable puts removing index rows for updated timestamped values?

call and is not related to how the client > makes the call right? I'm asking because we suspected timestamp issues and > we tried to introduce a delay between the initial put and the second one but > we had the same results. > > -GS > > On Mon, Mar 29, 2010 at 12:25 PM,

Re: IndexedTable puts removing index rows for updated timestamped values?

Def not the expected behavior and does not sound like user error.. Quick skim looks likk its https://issues.apache.org/jira/browse/HBASE-2286. Hbase does not gracefully handle the case where a put after a delete both have the same millisecond timestamp. Indexing table contrib was using this patter

Re: hbase-0.20.3 scan exception

Can you track down where that NPE is? (looks like somewhere on the regionserver). I the server stack should be in the logs.. -clint 2010/3/15 : > > Hi, > > using hbase-0.20.3 create transactional indexed table and when I scan table > insert more then 400,000 records it throws exception , > but i

Re: removing incomplete indexes

Is index.tableName the base table name? And index.indexName is the indexID that you passed to createIndexedTable? If so then looks like the metadata got corrupted. What does the IndexedTableDescriptor look like inside of removeIndex? (IndexedTableAdmin:111). If its gone in the meta, but the table

Re: Shared org.apache.hadoop.hbase.client.transactional.TransactionState?

nning to understand... > > So the state of the transaction (the puts, deletes, etc) is held in memory. > Is it held in the memory of the HBase Master? > > > > On Thu, Jan 28, 2010 at 1:10 PM, Clint Morgan >wrote: > > > The transactional contrib stuff is more geared to

Re: Shared org.apache.hadoop.hbase.client.transactional.TransactionState?

The transactional contrib stuff is more geared to short-duration transactions. It relys on OCC, and the optimistic assumption breaks down as the transaction gets longer (gives more time for another trx to put in a write which will cause a conflict and then the long trx will have to retry). Also, al

Re: Secondary indexes and transactions

AM, Mridul Muralidharan wrote: > Clint Morgan wrote: > >> After the 2PC process has determined that a commit should happen there is >> no >> roll-back. The commit must be processed. >> > > > From org.apache.hadoop.hbase.client.transactional.TransactionManager

Re: Secondary indexes and transactions

After the 2PC process has determined that a commit should happen there is no roll-back. The commit must be processed. So in your example, a commit has been approved, and one the of the regions is told to go ahead and commit. The region triggers the index Put, but then fails on his Puts (like out o

Re: Secondary indexes and transactions

The client drives the 2PC process, so after it has established that a trx may be committed (by asking each region), it tells each region to commit. Only then does it actually start to write to the base/indexed tables. So we don't really have a problem with "overlapping rollbacks", because a rollbac

Re: IndexedTableTest sample

A bit of config is needed. Look at the package javadoc in o.a.h.h.client.tableindexed. Have a look at TestTableIndexed for client usage. The following test made from your code works for me: package org.apache.hadoop.hbase.client.tableindexed; import java.io.IOException; import junit.framework.A

Re: Multiple puts with same timestamp

Yep, thanks for the pointer. On Tue, Nov 10, 2009 at 1:00 PM, stack wrote: > See HBASE-1485. Is this what you are asking about? > St.Ack > > On Tue, Nov 10, 2009 at 10:41 AM, Clint Morgan >wrote: > > > I have a couple questions about behavior of Puts with the same &g

Multiple puts with same timestamp

I have a couple questions about behavior of Puts with the same timestamps: If I do two puts with LATEST timestamp, and they both get added with the same value for now, will the last put be the visible one? If I do two puts with the same manual timestamp, will the last put be the visible one? I'l

Re: Region server going down

In your first post, you are hitting 1858. Fixed in trunk and 0.20 branch, but you will need to add the config value to recover from the WAL. I take issue with Ryan's handwavy statement about index/trx extensions not being scalable. With the indexing you pay an extra cost on puts which is essentia

Re: Optional memstore flush

below in a cron job? > > echo "flush 'TABLENAME'" | ./bin/hbase shell > > ... or variations thereof. > > St.Ack > > > On Wed, Sep 23, 2009 at 1:23 PM, Clint Morgan > wrote: > > > Is there no optional memstore flush anymore? I recall in 0.19 th

Optional memstore flush

Is there no optional memstore flush anymore? I recall in 0.19 the memcache would flush every so-often and you could configure this period (optional cache flush interval). Digging through now, I don't see it in 0.20. Is this mechanism no longer supported? Due to a couple of mixups, our stop cluste

Re: HBase cracks on EC2

rtell > wrote: > > > Good catch! It looks like there is more than one problem here. > > > > - Andy > > > > > > > > > > > > From: Clint Morgan > > To: hbase-user@hadoop.apache.org > >

Re: HBase cracks on EC2

This looks like the problem to me: java.io.IOException: wrong key class: org.apache.hadoop.hbase.regionserver.HLogKey is not class org.apache.hadoop.hbase.regionserver.transactional.THLogKey at Have you enable transactional hbase? Looks like you are using regular HBase to recover a THBase

Re: Row lock issues with Indexed Tables

I'm not sure if people have been using row locks with ITHBase. I havent been. So looks like you found an issue, and have a solution for it. Sweet. Create a JIRA for it and attach your patch. cheers, -clint On Mon, Sep 14, 2009 at 12:52 PM, Keith Thomas wrote: > > Apologies for the long post but

Re: Possible to set the results' sort method?

perhaps you would be interested in the tableindexed package. (Its in transactional contrib, see doc in o.a.h.h.client.tableindexed, or look at the tests). It will allow you to get a scanner whose results are ordered by a column's values (If you have an index on that column). -clint On Thu, Sep 1

Re: 0.20.0 migration

Nevermind, i figured it out: I did not have my hbase/hadoop site files in the hbase conf directory. On Wed, Jul 29, 2009 at 3:34 PM, Clint Morgan wrote: > I'm trying to migrate some data from 0.19 and experiencing some pain: > > I tried running HBaseAdmin.majorCompact() over

0.20.0 migration

I'm trying to migrate some data from 0.19 and experiencing some pain: I tried running HBaseAdmin.majorCompact() over all the tables, including catalogs. But when I try to migrate afterward I get error about root not being compacted. I had called majorCompact(-ROOT-) but never see it being compacte

Re: IndexedTable and Delete

Yeah, you've basically got it right. Its a bug. Please open a JIRA (and perhaps take a stab at a patch). Its low on my priority list as we mostly just do updates or delete whole rows.. -clint On Tue, Jul 21, 2009 at 1:04 PM, Andrew McCall wrote: > Hi, > > I've been using the IndexedTable stuf

Re: Transactions and Map/Reduce

Unfortunately, there is no more documentation on transactions. I've never used transactions in M/Rs. You are correct that if you only start/commit a transaction in mapper then you will only get atomicity across the individual map. One thing to keep in mind is that, in the current impl, all the wr

Re: Example of searching via index?

I'll add javadoc for IndexKeyGenerator for 0.20. -Read the package javadoc in client.tableindexed. -Have a look at the implementing class (SimpleindexKeyGenerator) for how its used. -Look at the test for more perspective. On Sun, Jun 21, 2009 at 10:52 PM, schnitzi wrote: > > I've created a table

Re: Delete all behaviour for IndexedTable...

DeleteAll should cascade to indexes (assuming you are running indexed regionserver). Please file a JIRA if you observe otherwise. -clint On Mon, May 25, 2009 at 1:01 AM, John Ellis (MELBOURNE-AU) < joh...@microsoft.com> wrote: > Hi - > > Will a call to deleteAll(byte[] row) on an IndexedTable ca

Re: HBase indexes

One thing to be sure is that you are starting up the IndexedRegionServer. See the package.html java doc in the client.tableindexed. This is what does the maintenance of the index tables. You can see if anything got in the indexes by doing a normal scan of the index table (EG scan 'foo-ip') -clint

Re: hbase.filter.ColumnValueFilter CompareOp EQUAL/NOT_EQUAL

That was a bug introduced in 0.19.2. Has since been fixed. ( https://issues.apache.org/jira/browse/HBASE-1264) Your expectations are correct; the behavior you describe is not... -clint On Wed, May 13, 2009 at 2:13 PM, Sasha Dolgy wrote: > Hi there, i'm a little confused, maybe I missed the plo

Re: cannot connect with hbase client

When starting up hbase, you need to tell it to spin up transactional region servers (impl, not just ipc interface) from hbase.client.transactional package.html javadoc: To enable transactions, modify hbase-site.xml to turn on the TransactionalRegionServer. This is done by setting hbase.regionse

Re: Secondary index

Yeah, thats a current limitation: indexes for a table must be pre-defined. I'm at a point where I need the ability to add/modify indexes for existing tables as well, so may work on a patch in the next week or so... -clint On Fri, Feb 20, 2009 at 2:15 PM, Slava Gorelik wrote: > Hi.From secondary

Re: Secondary Indexes problem.

Yes, that looks like the right way to fix it. Quick fix to make it work now, I bet you can just remove that hbase.regionserver.class property (or set it to the TransactionalInterface). I see now thats what I had been doing in my deployment. Care to submit a patch with the real fix? -clint On Fr

Re: can tableindexed be modified afterward?

Currently indexes must be declared prior to table creation. It would not be too hard to add this feature though... -clint On Mon, Dec 22, 2008 at 7:21 PM, Rong-en Fan wrote: > According to list archive and tableindexed documents, it seems to me that > the index can only be added when the table

Re: Unique constraint

You could maintain an index, and then use the index to jump right to rows for the column value(s) in question. See the tableindexed package for more info.. cheers, -clint On Thu, Dec 18, 2008 at 4:37 AM, Slava Gorelik wrote: > Hi.I'm trying to implement Unique constraint on some column, but it l

Re: Integer key range scan

--- > > From: clint....@gmail.com [mailto:clint@gmail.com] On Behalf Of > > Clint Morgan > > Sent: Tuesday, December 16, 2008 2:41 PM > > To: hbase-user@hadoop.apache.org > > Subject: Re: Integer key range scan > > > > Actually, thats not the case: &g

Re: Integer key range scan

the JVM is using for an integer you have to > write > your own function that uses bitwise operations to extract the value into a > byte[]. > > Anyways, the answer is Yes. > > JG > > > -Original Message- > > From: clint@gmail.com [mailto:clint@gmail.

Re: Integer key range scan

Does intToBytes preserve ordering? IE is the leicographic ordering of the resulting byte [] the same as the ordering of the original ints? I think Edward needs that for his scan to work... -clint On Tue, Dec 16, 2008 at 1:25 PM, Jonathan Gray wrote: > I believe there are intToBytes/bytesToInt

Re: Loosened transaction isolation level

herwise, what's the plan? Leave it as is and fix it once we have zk? > > Thx! > > On Mon, Oct 27, 2008 at 9:55 AM, Clint Morgan <[EMAIL PROTECTED]> > wrote: > > > Yeah, the scanner's are definitely overly conservative. I needed this > > because for ex

Re: Loosened transaction isolation level

Yeah, the scanner's are definitely overly conservative. I needed this because for example I am scanning a key range to guarantee that the update I'm processing has a unique column value. The current implementation could be improved to look more carefully if the scan really did pass any rows which w

Re: Agenda for tomorrow's meeting

Hey Stack, I'd like to dial in to this. Let me know what I need to do... cheers, -clint On Mon, Oct 20, 2008 at 11:01 AM, stack <[EMAIL PROTECTED]> wrote: > I was thinking that we could make it a working session since it looks like > the bulk of attendees have running hbase instances and so hav

Re: Pigi project

You could use https://issues.apache.org/jira/browse/HBASE-883 for that query too (and <, >, as well). On Fri, Oct 17, 2008 at 4:44 AM, Krzysztof Gałęcki < [EMAIL PROTECTED]> wrote: > Hi. > > Using Pigi, you can execute such query (but not query with column1>n or > column1 have > some troubles wit

Re: [VOTE] hbase 0.18.1 release candidate 1

I was hoping someone could take a look at 910. It seems like a critical issue to me. On Thu, Oct 16, 2008 at 2:12 PM, stack <[EMAIL PROTECTED]> wrote: > hbase 0.18.1 release candidate 1 is available for download: > > > http://people.apache.org/~stack/hbase-0.18.1-candidate-1/

Re: scanning with custom filter problem

I'm using a couple of custom filters. I seem to recall a similar initial problem with them being serialized, and it seems like the errors were being swallowed and not making it to the logs (as Stack mentioned). I think it was something silly on my part like a missing default constructor, but can't

Re: Other CC examined in HBase transaction model?

r > that > in 0.18.0, is that still > work in progress? > > Thanks a lot! > > On Mon, Oct 6, 2008 at 2:26 PM, Clint Morgan <[EMAIL PROTECTED]> wrote: > > > I implemented the OCC functionality. Went that route because it seemed > > relatively straightforward a

Re: Other CC examined in HBase transaction model?

I implemented the OCC functionality. Went that route because it seemed relatively straightforward and has acceptable performance for our workloads. Also I saw something somewhere that made me think that google was using OCC for transactions in AppEngine. My intuition is that where the OCC approach

Re: hbase slow to startup on copy of hbase directory

> I'll open a JIRA. Nevermind, I see you guys are one step ahead: https://issues.apache.org/jira/browse/HBASE-730

Re: hbase slow to startup on copy of hbase directory

Thanks for the response, that sounds good. I had a quick peek at the code, but I don't understand what is going on there well enough to implement the proposed solution... I'll open a JIRA. cheers, -clint On Tue, Jul 8, 2008 at 9:14 AM, Jim Kellerman <[EMAIL PROTECTED]> wrote: >> -Original Me

hbase slow to startup on copy of hbase directory

Hi all, I'm having a little problem with our tests that use hbase. First, I run a test which generate all of the hbase tables, and exits. Then for each test, I copy over the hbase directory, and the start up hbase. So far, so good, hbase quickly starts up and finds all my tables. However, I the

Re: HTable.deleteAll(byte[] row) - doesn't work

Perhaps the problem is that you are referring to row keys as string in the put, and as byte[] in the delete. The string argument put method uses UTF_8 encoding, String.getBytes() uses the default encoding which may not be the same... -clint On Sun, Jun 22, 2008 at 12:47 PM, acure <[EMAIL PROTECT

Re: Problem with scanner again

onserver A's state? > > Thanks, > St.Ack > > Clint Morgan wrote: >> >> Why the aversion to filters? Thats how we solve this problem, I have a >> simple SkipRowFilter that I wrote that does exactly this... >> -clint >> >> 2008/6/18 Krzysztof Gałę

Re: Problem with scanner again

Why the aversion to filters? Thats how we solve this problem, I have a simple SkipRowFilter that I wrote that does exactly this... -clint 2008/6/18 Krzysztof Gałęcki <[EMAIL PROTECTED]>: > I can't cache items on List because of two main reasons: > > 1. I have to many items to cache them in memory

Question about hlog recovery.

I have a local cluster running, and its logging to /log_X.X.X.X_1213228101021_60020/ Then I kill both master and regionserver, and restart. Looking through the logs I don't see anything about trying to recover from this hlog, it just creates a new hlog alongside the existing one (with a new startc

Re: hbase 0.1.3 release candidate 1

We use getRow() everytime we want a single row (we almost always want multiple columns). We're running a patched version of TRUNK with light development usage, and have not seen any issues. So getRow() is important for us, but we won't use 0.1 so won't pollute the vote... -clint On Tue, Jun 10,

Re: Does HBase support single-row transaction?

> "When the application creates an entity, it can assign another entity as the > parent of the new entity. Assigning a parent to a new entity puts the new > entity in the same entity group as the parent entity." > > I think I need to sign up for app engine and use it to see if I can figure > how th

Re: Does HBase support single-row transaction?

08 at 2:13 PM, stack <[EMAIL PROTECTED]> wrote: > Clint Morgan wrote: >> Zookeeper makes good sense for distributed locking to get isolation. >> But we still need transaction start, commit, and rollback to get >> atomicity. I think this properly belongs in hbase. >>

Re: Does HBase support single-row transaction?

Responses inline: 2008/5/27 Bryan Duxbury <[EMAIL PROTECTED]>: > It seems like if you wanted to do some manner of multi-row transactional > put, the only real way to manage it is with deletes. That is, if the first > put succeeds but the second fails, you can "invert" the first put into a > bunch

Re: Does HBase support single-row transaction?

Zookeeper makes good sense for distributed locking to get isolation. But we still need transaction start, commit, and rollback to get atomicity. I think this properly belongs in hbase. So suppose I want to read two rows, and then update them as an isolated, atomic action: try { getZookeeperLock

Re: Compound filters

https://issues.apache.org/jira/browse/HBASE-583 is about the column matching part. Then you could use RowFilterSet to get AND or OR. -clint On Wed, May 21, 2008 at 11:07 AM, David Alves <[EMAIL PROTECTED]> wrote: > Hi Guys > >I'm currently needing to build some compound filters for colum

Re: Feedback on my implementation.

On Thu, May 8, 2008 at 3:35 PM, Josh <[EMAIL PROTECTED]> wrote: > To allow matching on column values, I have put data into the column > family attribute name, for example: Have a look at the filter interface, RegexpRowFilter will let you match rows whose columns have certain values. Sound like

Re: hbase on ec2 with s3 anyone?

I've tried s3 on hbase. See a previous post I made on this list. Basically its considerably slower than hdfs, especially so for random reads. Also I think there could be consistency issues when running s3: master creates file, tells region server to read it, and region server gets a file not found.

Re: comparing hbase backed by HDFS verses S3

keep hbase responsive by reading from s3 while hdfs is down. While this does not provide a consistent snapshot at a given point in time like HBASE-50 is asking, I think it would work for us as a data backup. cheers, -clint On Thu, May 1, 2008 at 5:14 PM, Clint Morgan <[EMAIL PROTECTED]>

Re: comparing hbase backed by HDFS verses S3

. Alternatively, maybe you can convince distcp to go between > hdfs:// and http://? Hbase files should never really be bigger than a 1G or > so, so it should be 'safe'. > > St.Ack > > > > Clint Morgan wrote: > > > Thanks for the input as it confirmed my

Re: comparing hbase backed by HDFS verses S3

xpecting this cluster to live forever, maybe you > should keep your authoritative data on s3 (native or s3fs) and just reload > HBase on cluster init? > > ckw > > Chris K Wensel > [EMAIL PROTECTED] > http://chris.wensel.net/ > http://www.cascading.org/ > > >

comparing hbase backed by HDFS verses S3

We are considering using S3 as the DFS impl for hbase. I ran some benchmarks to get an idea for the performance differences. We are particularly interested in being able to serve data to users from hbase, so want low latency responses for getting 10s of rows. Each row ("transaction") has about 1K

Re: Lost Rows

On Wed, Apr 23, 2008 at 11:58 AM, David Alves <[EMAIL PROTECTED]> wrote: > On another matter must hbase really log (even in debug) all filter > calls? Thats stands for about 70% of my logs. Agreed, I'll drop those messages to trace.

Re: Secondary indexes

ies that might > cover more than a "few" regions - can you imagine having to check 100 or > 1000 regions for the next result every time you needed to iterate? > > > > On Apr 22, 2008, at 12:58 PM, Clint Morgan wrote: > > > > Yeah, that would be an easy approach

Re: Secondary indexes

would solve that > problem. > > -Bryan > > > > On Apr 22, 2008, at 11:58 AM, stack wrote: > > > > Some questions interlaced below: > > > > Clint Morgan wrote: > > > > > All, > > > > > > We want to put secondary inde

Re: Secondary indexes

> I don't follow what the Factory adds. The Factory part of the name just means that it can make an object of type T from a byte[] (the column value). This is the type that we keep in the set and sort on. > > We're talking about getting HBASE-82 into 0.2. Does that interfere with > this propos

Secondary indexes

All, We want to put secondary indexes into hbase. The motivation is that we are storing data in hbase that we want to serve to users. We would like to be able to serve rows sorted by column values. Our queries will be over rows with a given key prefix, so we should not be hitting to many regions.

Re: StackOverFlow Error in HBase

Try the patch at https://issues.apache.org/jira/browse/HBASE-554. cheers, -clint On Mon, Mar 31, 2008 at 5:39 AM, David Alves <[EMAIL PROTECTED]> wrote: > Hi ... again > > In my previous mail I stated that increasing the stack size solved the > problem, well I jumped a little bit to the

Re: Filter omitting columns

ue got filtered ? > > Is there a tutorial available that shows 2-3 tables in RDBMS style and > then transforms them into column oriented design ? > > Thanks > -Ankur > > > > > > > -Original Message- > From: Clint Morgan [mailto:[EMAIL PROTECT

Re: Filter omitting columns

> 'familyname:' as the column. This creates a wild-card scanner > which will do what you want without filters. > > --- > Jim Kellerman, Senior Engineer; Powerset > > > > > > -Original Message- > > From: Clint Morgan [mailto:[EMAIL PROTEC

Re: Filter omitting columns

I was having a similar problem as well. Though I've never used just the column families to specify the columns (eg always fully qualified col names like family:col) . Maybe you can try my patch and see if it fixes your problem. https://issues.apache.org/jira/browse/HBASE-527 Also you can give a n