Hi Marcos,
Here's a JIRA query that will show you the list of unresolved Hive issues
related to the HBase storage handler:
https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&jqlQuery=component+%3D+%22HBase+Handler%22+AND+project+%3D+HIVE+AND+resolution+%3D+Unresolved+O
El 4/12/2011 2:13 PM, Jean-Daniel Cryans escribió:
Is there anything in particular you'd like to know?
I recently answered a more specific (but still general) question about
Hive/HBase here: http://search-hadoop.com/m/YZe7h1zxxoc1
I will also be giving a presentation at OSCON Data in July about
The simplest thing to do would be to just store the row key from TableMain
as the column qualifier in the TableRLookup row. Then you don't even need
to append to an existing value, you're just adding a new column where the
qualifer stores the row ID and the value can be empty. So instead of using
J-D, Thanks I commented out this part as it's trying to limit based on
the block size, which in this case is set artificially high to
guarantee only one block.
On Tue, Apr 12, 2011 at 4:43 PM, Jean-Daniel Cryans wrote:
> The namenode does this:
>
> long remaining = node.getRemaining()
Sounds like a job for a coprocessor; the cell-append-coprocessor. You
load it on a particular column family and anything put to this
processor appends to the existing cell value, if a value already
exists.
St.Ack
On Tue, Apr 12, 2011 at 8:54 AM, Vishal Kapoor
wrote:
> rebuilding the whole revers
Ah!! I always forget to check the region server log:
java.io.IOException: Compression algorithm 'lzo' previously failed test.
at
org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:77)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.jav
See in the below.
On Tue, Apr 12, 2011 at 4:38 AM, Michael G. Noll
wrote:
> So in order to help myself and hopefully also other readers of this
> mailing list, I try to summarize my steps so far to understand and build
> Hadoop 0.20-append for use with HBase 0.90.2, the problems I have run
> into
No, all I say is that you don't need the METHOD, eg this will work in
both 0.20 and 0.90:
>alter 'cjjHTML', {NAME => 'responseHeader'}
J-D
On Tue, Apr 12, 2011 at 4:39 PM, 陈加俊 wrote:
> So there is no method to add family for a table?
> I need to drop the table and create it If I want to add ano
The namenode does this:
long remaining = node.getRemaining() -
(node.getBlocksScheduled() * blockSize);
// check the remaining capacity of the target machine
if (blockSize* FSConstants.MIN_BLOCKS_FOR_WRITE>remaining)
MIN_BLOCKS_FOR_
So there is no method to add family for a table?
I need to drop the table and create it If I want to add another family to a
table ? How to keep the data in the table before?
On Wed, Apr 13, 2011 at 2:31 AM, Jean-Daniel Cryans wrote:
> No need to specify the method for adding, it wasn't required
Hmm... There's no physical limitation, is there an artificial setting?
On Tue, Apr 12, 2011 at 4:27 PM, Jean-Daniel Cryans wrote:
> It says:
>
> 2011-04-12 16:16:17,157 DEBUG [IPC Server handler 7 on 51372]
> namenode.ReplicationTargetChooser(408): Node
> /default-rack/127.0.0.1:22967 is not chos
It says:
2011-04-12 16:16:17,157 DEBUG [IPC Server handler 7 on 51372]
namenode.ReplicationTargetChooser(408): Node
/default-rack/127.0.0.1:22967 is not chosen because the node does not
have enough space
J-D
On Tue, Apr 12, 2011 at 4:24 PM, Jason Rutherglen
wrote:
> Ah, I had changed conf/log4j
Ah, I had changed conf/log4j.properties. So I changed
src/test/resources/log4j.properties, and now the -output file's much
more verbose. I'm not sure I understand what's going on however.
I'll try to make sense out of the log:
http://pastebin.com/MrQJcbJr
On Tue, Apr 12, 2011 at 3:38 PM, Stack
Yep.
J-D
On Tue, Apr 12, 2011 at 3:55 PM, Chris Tarnas wrote:
> I was looking at the metrics column in the regionservers web UI and had a
> question:
>
> If I understand correctly hbase.hregion.max.filesize is the max size of a
> single column family's storefile size. If I have 4 column famili
I was looking at the metrics column in the regionservers web UI and had a
question:
If I understand correctly hbase.hregion.max.filesize is the max size of a
single column family's storefile size. If I have 4 column families in a table
and have hbase.hregion.max.filesize set to 1GB that means I
Yes there are a few places like that. Also when you create new
HTables, you should also close their connections (this is not done in
htable.close).
See HTable's javadoc which says:
Instances of HTable passed the same Configuration instance will share
connections to servers out on the cluster and
You changed the src/test/resources/log4j.properties?
Not sure why changing the block size would make a difference, why it
would even care.
St.Ack
On Tue, Apr 12, 2011 at 2:38 PM, Jason Rutherglen
wrote:
> Thanks, I'm only seeing the error when I change the block size, either
> via DFSClient.cre
Could you upgrade to the newly released CDH3 instead? It has a few more fixes.
So regarding your issue, I don't see regions stuck. The first one did
timeout on opening but then it was reassigned (and then I can't see
anything in the log that says it timed out again).
By the way can you check what
In the middle of upgrading our dev environment from 0.89 to 0.90.2CDH3B4.
When we did the upgrade locally (Macs), no issues came up. Different story
on our EC2 dev box it seems.
Background:
- dev is running in pseudo-cluster mode
- we neglected to set replication to 1 from 2 the first time we star
I'm running into the same issue, but did some poking around and it seems that
Zookeeper connections are being left open by an HBase internal.
Basically, I'm running a mapreduce job within another program, and noticed in
the logs that every time the job is run, a connection is open, but I never s
Thanks, I'm only seeing the error when I change the block size, either
via DFSClient.create or via the Configuration dfs.block.size property.
When I changed the log4j.properties to
'log4j.logger.org.apache.hadoop=WARN' I'm not seeing anything
additional in the output in the target/surefire-reports
At first sight, it seems to be a network problem. There were some datanodes
that were un-reachable from the HBase regionserver, as evidenced by the
following logs:
53799733 from any node: java.io.IOException: No live nodes contain current
block. Will get new block locations from namenode and retry
It's more in the vain of
https://issues.apache.org/jira/browse/HBASE-3755 and
https://issues.apache.org/jira/browse/HBASE-3771
Basically 0.90 has a regression regarding the handling of zookeeper
connections that make it that you have to be super careful not to have
more than 30 per machine (each n
Depends what the log4j.properties file that your code is picking up says.
mvn test or IDE "run" test classes should pick up
src/test/resources/log4j.properties, which will log to stderr. If that's
how you're running you could tweak the hadoop logging level to see if it
shows anything more. Change
Where does MiniDFSCluster store the logs? I don't see a location,
assuming it's different than stdout/err.
On Tue, Apr 12, 2011 at 11:26 AM, Stack wrote:
> The datanodes are not starting? Anything about that in the log?
> St.Ack
>
> On Tue, Apr 12, 2011 at 11:13 AM, Jason Rutherglen
> wrote:
>
YouAreDead means that the master is already processing the death of
those region servers when the region server talks back to the master.
Network split?
J-D
On Tue, Apr 12, 2011 at 11:33 AM, Vidhyashankar Venkataraman
wrote:
> This was something that happened a week back in our cluster: There wa
You can also cat the .regioninfo file that is under the region
directory to learn more about the region -- its HRegionInfo (The file
format is serialized HRegionInfo followed by a String version so
intelligible to non-machines).
St.Ack
2011/4/12 Stack :
> These regions that are in hdfs but not in
These regions that are in hdfs but not in .META. and not on any region
server are probably harmless. Would be interesting to trace how they
got to this state. My guess is that they are old let go regions that
were not cleaned up.
The region directory name is its encoded name which is the tail pa
This was something that happened a week back in our cluster: There was a flash
death of region servers: a few of the region servers did have near-full heaps
so I thought GC could be at play. But many of them crashed after a few DFS
errors followed by a YouAreDeadException and they didn't have GC
No need to specify the method for adding, it wasn't required in 0.20.6
either so if it was accepting it it was a bug.
J-D
On Tue, Apr 12, 2011 at 1:54 AM, 陈加俊 wrote:
> I can add family by follow command In HBase-0.20.6
>
>>alter 'cjjHTML', {NAME => 'responseHeader', METHOD => 'add'}
>
> But in
That usually means that your datanode refuses to start or isn't able
to connect for some reason. Have a look at its log.
J-D
On Tue, Apr 12, 2011 at 11:13 AM, Jason Rutherglen
wrote:
> I'm running into an error when setting the DFS block size to be larger
> than the default. The following code
The datanodes are not starting? Anything about that in the log?
St.Ack
On Tue, Apr 12, 2011 at 11:13 AM, Jason Rutherglen
wrote:
> I'm running into an error when setting the DFS block size to be larger
> than the default. The following code is used to create the test
> cluster:
>
> Configuratio
I'm running into an error when setting the DFS block size to be larger
than the default. The following code is used to create the test
cluster:
Configuration conf = new Configuration();
MiniDFSCluster cluster = new MiniDFSCluster(conf, 2, true, null);
FileSystem fileSys = cluster.getFileSystem();
On Tue, Apr 12, 2011 at 7:28 AM, 茅旭峰 wrote:
> Hi,
>
> I've noticed that Cloudera has announced the CDH3 release, but the apache
> hbase 0.90.2 is also just released.
All should upgrade to the CDH3 release. It includes hdfs-1520,
hdfs-1555, and hdfs-1554 -- important features not present in
CDH3
Is there anything in particular you'd like to know?
I recently answered a more specific (but still general) question about
Hive/HBase here: http://search-hadoop.com/m/YZe7h1zxxoc1
I will also be giving a presentation at OSCON Data in July about our
experience using both together.
J-D
On Mon, Ap
On Tue, Apr 12, 2011 at 12:30 AM, Oleg Ruchovets wrote:
> Hi
> We running on hbase 0.90.1 and want to upgrade to 0.90.2.
> What is the process to do it , I mean is it enough to change the jar or it
> requires configuration files changes.
Its a point release so stop the cluster, replace your hba
So, this is a special case that I allow in the manual. I should add a
note, "Pseudo-distributed using an external zk ensemble?". Would that
do?
Thanks Stanley,
St.Ack
2011/4/12 :
> Thanks St.Ack;
> I am not sure if this is right.
> Currently I set hbase.cluster.distributed to be true, and set
On Tue, Apr 12, 2011 at 10:52 AM, Robert Gonzalez
wrote:
> Ok, deleted logs that master was complaining about, restarted master
> only. Seemed to be stable after a bunch of the messages like the one
> below, then restarted regionservers, sans the one that gave me trouble
> this morning. Now seem
Does the RegionServer OOME opening same file each time it happens?
If so, something is up with the file. Move it aside to get your
cluster online and then lets try and figure whats in this file that
brings on OOME -- a very large record or perhaps a corruption?
St.Ack
On Mon, Apr 11, 2011 at 9:
Ok, deleted logs that master was complaining about, restarted master
only. Seemed to be stable after a bunch of the messages like the one
below, then restarted regionservers, sans the one that gave me trouble
this morning. Now seems to be up and running again. I don't trust
it, seen this kind of
Please paste more of your log. Your .META. table has empty cells for
startcode for some reason but the log below is just WARNING you of
this fact and skipping those cells.
St.Ack
On Tue, Apr 12, 2011 at 10:40 AM, Robert Gonzalez
wrote:
> A bunch of this in the master log:
>
> 2011-04-12 12:38:2
Its up in the release repository now. Let me know if any issue with
it (I'm still not sure how I made it happen -- I need to do this
process again soon).
Thanks for your patience,
St.Ack
On Mon, Apr 11, 2011 at 8:25 AM, Stack wrote:
> Lior and Joe:
>
> Sorry for the mvn lag. The mvn deploy syst
A bunch of this in the master log:
2011-04-12 12:38:23,771 WARN
org.apache.hadoop.hbase.master.CatalogJanitor: REGIONINFO_QUALIFIER is
empty in
keyvalues={urlhashcopy,E3208173766FDD7C01FE9633E281ED0A,1296085183252.7501ae2b7e933057ea12610c4ec6d001./info:server/1296142856167/Put/vlen=41,
urlhashcop
Robert,
You can stop the daemons individually on each node:
bin/hbase-daemon.sh stop master
bin/hbase-daemon.sh stop regionserver
Use this to stop the processes that can be cleanly shutdown. Then let's
look at which processes are still hanging and what the logs of the hanging
processes are show
In another terminal, you can see which region server(s) is still running.
Checking region server log on that region server would help.
On Tue, Apr 12, 2011 at 10:34 AM, Robert Gonzalez <
g...@maxpointinteractive.com> wrote:
> You mean like this:
>
> hbase@c1-m02:/usr/lib/hbase-0.90.0/bin$ ./stop-
You mean like this:
hbase@c1-m02:/usr/lib/hbase-0.90.0/bin$ ./stop-hbase.sh
stopping
hbase...
Agreed. I've seen similar issues when upon startup where for whatever
reason an hlog (often empty) can't be read, which hangs the startup
process. Manually deleting it from HDFS clears the issue.
On Tue, Apr 12, 2011 at 10:01 AM, Jinsong Hu wrote:
> You probably should stop all master/regionserv
You probably should stop all master/regionservers, then start one master,
tail -f the log to confirm all the hlogs are handled,
then start the first regionserver, and then other regionservers.
I have encountered this issues before.
hbase is not as good as what you want, but not as bad as you sa
We've been using HBase for about a year, consistenly running into
problems where we lost data. After reading forums and some back and
forth with other Hbase users, we changed our data methodology to save
less data per row. This last time, we upgraded to 0.90 at the
recommendation of the hbase com
The plan was to have the map reduce job run on our schedule (hourly, daily,
monthly) and populate these rollups so we aren't having to do any processing
on the data in hbase. When a user requests stats, we just pull back the
already compiled data from the rollups. It isn't realtime this way, but w
We've been using HBase for about a year, consistenly running into problems
where we lost data. After reading forums and some back and forth with other
Hbase users, we changed our data methodology to save less data per row.
This last time, we upgraded to 0.90 at the recommendation of the hbase
comm
Have you read the thread entitled 'min, max' ?
On Tue, Apr 12, 2011 at 7:33 AM, Vishal Kapoor
wrote:
> Here is the problem.
>
> my row Ids are "starting" with reversed time stamp followed by "/" and
> some more values.
>
>
> 9223370735421724555/TimeStamp1/TimeStamp2/CustomerId/MacIdSystem1/MacIdS
It depends on how many concurrent users on the BI frond end, if each of them
will fire off a MR job for their BI queries, which likely resulting in a scan
or partial scan on HBase, this may put too much stress on the IO sub-system.
If you have the data access pattern of your BI users, you may
rebuilding the whole reverse lookup table should be expensive if I am
looking for a million new rows every day in the master table,
reading a row manually and then writing the appended row should be a
solution but will be a pain.
for a file backed system doing a append should be possible?
Vishal
You have the keys for both tables, is there any reason you can't do a get,
local append, put?
If you do it in batch, then running a reduce job that collects all of the keys
for a given value would be fairly efficient.
Dave
-Original Message-
From: Vishal Kapoor [mailto:vishal.kapoor...
El 4/12/2011 10:19 AM, Shahnawaz Saifi escribió:
1- What HBase version are you use?
hbase-0.90.1
2- Which O.S are you using?
CentOS 5.5 x8_64
3- Which are the permitions on those directories:
/hbase/unassigned
/hbase/rs
/hbase/table
These tables/dir doesn't exist.
As I understood the
Do we have any API which can append text values or row Ids to a cell.
I want to do a control break report and want to append row Ids to a
cell value...
here is an example.
TableMain
row1 : user1
row2: user1
row3 : user2
row4 : user3
reverse lookup.
TableRLookup.
user1 : row1,row2
user2: row3
us
I get this occasionally..(not all the time)..Upgrading from 0.20.6 to 0.90.2
Is this issue same as this JIRA
https://issues.apache.org/jira/browse/HBASE-3578
I'm using HBaseConfiguration.create() & setting that in job
thx
v
2011-04-12 02:13:06,870 ERROR Timer-0
org.apache.hadoop.hbase.mapre
Here is the problem.
my row Ids are "starting" with reversed time stamp followed by "/" and
some more values.
9223370735421724555/TimeStamp1/TimeStamp2/CustomerId/MacIdSystem1/MacIdSystem2/RowType
the RowId is designed to make sure the latest row comes up first in the Scan.
reverse time is calc
Hi,
I've noticed that Cloudera has announced the CDH3 release, but the apache
hbase 0.90.2 is also just released.
I think both of them could run smoothly on CDH3 hadoop 0.20.2+923.21, but
any difference there?
Thanks and regards,
Mao Xu-Feng
I start a new thread to make things clearer.
It seems like we have run into a inconsistent table issue.
We are using hadoop-0.20.2-CDH3B4, zookeeper-3.3.2-CDH3B4
and hbase-0.90.1-CDH3B4.
The original issue we've run into is like
hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123
1- What HBase version are you use?
hbase-0.90.1
2- Which O.S are you using?
CentOS 5.5 x8_64
3- Which are the permitions on those directories:
/hbase/unassigned
/hbase/rs
/hbase/table
These tables/dir doesn't exist.
As I understood these are the INFO level logs. But curious to know the
El 4/12/2011 8:09 AM, Shahnawaz Saifi escribió:
2011-04-12 06:52:37,784 INFO
org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
KeeperException when processing sessionid:0x12f48ec06850367 type:create
cxid:0x2 zxid:0xfffe txntype:unknown reqpath:n/a Error
Path:/hbase/unas
2011-04-12 06:52:37,784 INFO
org.apache.zookeeper.server.PrepRequestProcessor: Got user-level
KeeperException when processing sessionid:0x12f48ec06850367 type:create
cxid:0x2 zxid:0xfffe txntype:unknown reqpath:n/a Error
Path:/hbase/unassigned Error:KeeperErrorCode = NodeExists for
/hba
Hi all,
as a few other people on this mailing list I am currently working on
getting HBase up and running on Hadoop 0.20.2. I think I have by now
read most of the relevant past discussions on this topic here, e.g.
St.Ack's thread on creating an append release [6], Mike Spreitzner's
recent attempt
My guess was the second region is closed, but its directory named of the
encoded name is still
on the hdfs, this leads to the inconsistency.
I wonder what happened when the region is closed, it looks like the process
is over, from the log.
2011/4/12 茅旭峰
> Anyway, looks like the two regions bel
I can add family by follow command In HBase-0.20.6
>alter 'cjjHTML', {NAME => 'responseHeader', METHOD => 'add'}
But in HBase-0.90.2 I can't do it . How ?
--
Thanks & Best regards
jiajun
Hi
We running on hbase 0.90.1 and want to upgrade to 0.90.2.
What is the process to do it , I mean is it enough to change the jar or it
requires configuration files changes.
Please advice
Oleg.
Thanks St.Ack;
I am not sure if this is right.
Currently I set hbase.cluster.distributed to be true, and set the
HBASE_MANAGES_ZK=false; the environment is running now.
Both the hbase master and region server are running on the same machine. I
think from the definition, this is also a "pseudo-dis
69 matches
Mail list logo
