[jira] [Created] (HBASE-11341) ZKProcedureCoordinatorRpcs should respond only to members
Matteo Bertozzi created HBASE-11341:
---
Summary: ZKProcedureCoordinatorRpcs should respond only to members
Key: HBASE-11341
URL: https://issues.apache.org/jira/browse/HBASE-11341
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3, 0.94.20, 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.99.0, 0.94.21, 0.98.4
ZKProcedureCoordinatorRpcs nodeCreated() responds to events of every znode
under the procedure and not only members.
{code}
INFO: Received created event: /hbase//reached/
INFO: Received created event: /hbase//reached//
{code}
the result is a warning (no other side effects)
{code}
WARN [main-EventThread] procedure.ProcedureCoordinator: Member 'cluster_test'
is trying to release an unknown procedure 'reached
{code}
[~apurtell] posted in HBASE-10926 the steps to reproduce it:
{noformat}
$ cd ./src/hbase
$ git checkout master
$ mvn -DskipTests clean install
{noformat}
In one console:
{noformat}
$ ./bin/hbase master start
{noformat}
In another console:
{noformat}
$ ./bin/hbase org.apache.hadoop.hbase.util.LoadTestTool -num_keys 10 \
-read 10:100 -write 1:100:10
{noformat}
In a third console:
{noformat}
$ ./bin/hbase shell
hbase> while true do ; flush 'cluster_test' ; sleep 10 ; end
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11342) The method isChildReadLock in class ZKInterProcessLockBase is wrong
Qianxi Zhang created HBASE-11342:
Summary: The method isChildReadLock in class
ZKInterProcessLockBase is wrong
Key: HBASE-11342
URL: https://issues.apache.org/jira/browse/HBASE-11342
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.98.3, 0.99.0
Reporter: Qianxi Zhang
Assignee: Qianxi Zhang
Priority: Minor
The method isChildReadLock in class ZKInterProcessLockBase may be wrong, which
determines whether the lock is readLock or not. So we should compare the node
name with READ_LOCK_CHILD_NODE_PREFIX rather than WRITE_LOCK_CHILD_NODE_PREFIX.
Since there is no other method to invoke the method "isChildReadLock" now, we
have not encountered an error.
{code}
protected static boolean isChildReadLock(String child) {
int idx = child.lastIndexOf(ZKUtil.ZNODE_PATH_SEPARATOR);
String suffix = child.substring(idx + 1);
return suffix.startsWith(WRITE_LOCK_CHILD_NODE_PREFIX);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11343) Writing the RootRegionLocation znode in thrift format.
Manukranth Kolloju created HBASE-11343: -- Summary: Writing the RootRegionLocation znode in thrift format. Key: HBASE-11343 URL: https://issues.apache.org/jira/browse/HBASE-11343 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Minor Fix For: 0.89-fb -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11056) Make multiget work more like batchMutate
[ https://issues.apache.org/jira/browse/HBASE-11056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-11056. -- Resolution: Duplicate Duplicate of HBASE-8362. > Make multiget work more like batchMutate > > > Key: HBASE-11056 > URL: https://issues.apache.org/jira/browse/HBASE-11056 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Nick Dimiduk > > Right now, the multiget path issues a sequence of Region.get calls for the > single RPC. This is good, but it could be better. Multiple Puts are batched > together and applied as a group, I think we can do the same for Gets. The > idea is to reuse scanners for all Gets in the batch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11344) Hide row keys and such from the web UIs
Devaraj Das created HBASE-11344: --- Summary: Hide row keys and such from the web UIs Key: HBASE-11344 URL: https://issues.apache.org/jira/browse/HBASE-11344 Project: HBase Issue Type: Improvement Reporter: Devaraj Das Fix For: 0.99.0 The table details on the master UI lists the start row keys of the regions. The row keys might have sensitive data. We should hide them based on whether or not the user accessing has the required authorization to view the table.. To start with, we could make the display of row keys and such based on a configuration being true or false. If it is false, such potentially sensitive data is never displayed on the web UI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11345) Add an option not to restore cluster after an IT test
Jimmy Xiang created HBASE-11345: --- Summary: Add an option not to restore cluster after an IT test Key: HBASE-11345 URL: https://issues.apache.org/jira/browse/HBASE-11345 Project: HBase Issue Type: Test Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor After some IT test failure, we have to dig from logs to see what's wrong if we restore the cluster. It will be good if we can keep the current state of the cluster for investigation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11346) Enable cacheonwrite for index and bloom blocks by default
Nick Dimiduk created HBASE-11346: Summary: Enable cacheonwrite for index and bloom blocks by default Key: HBASE-11346 URL: https://issues.apache.org/jira/browse/HBASE-11346 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.99.0, 0.98.4 Reporter: Nick Dimiduk Theory is, this will reduce the cost of compactions on a mixed workload cluster, reducing the number of seeks for the first blocks from the new HFile from ~4 to ~1. -- This message was sent by Atlassian JIRA (v6.2#6252)
Metrics units?
Heya, Do we have documentation someplace describing the units for various metics? For instance, it's not always clear if a unit is bytes or milliseconds. Would be nice if the metric names were self-identifying, but that will break compatibility with anyone who's monitoring specific metrics by name, so maybe just an update to the book? -n
[jira] [Created] (HBASE-11347) For some errors, the client can retry infinitely
Nicolas Liochon created HBASE-11347: --- Summary: For some errors, the client can retry infinitely Key: HBASE-11347 URL: https://issues.apache.org/jira/browse/HBASE-11347 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.3, 0.99.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Critical Fix For: 0.99.0, 0.98.4 As the title says... -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Metrics units?
most of the metrics should have a description string in JMX that should describe them. If they are missing units in there and it's not clear, just let me know and I can try and clean that up. On Fri, Jun 13, 2014 at 10:46 AM, Nick Dimiduk wrote: > Heya, > > Do we have documentation someplace describing the units for various metics? > For instance, it's not always clear if a unit is bytes or milliseconds. > Would be nice if the metric names were self-identifying, but that will > break compatibility with anyone who's monitoring specific metrics by name, > so maybe just an update to the book? > > -n >
[jira] [Resolved] (HBASE-8315) Documentation should have more information of LRU Stats
[
https://issues.apache.org/jira/browse/HBASE-8315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-8315.
--
Resolution: Fixed
Fix Version/s: 0.99.0
Committed small text change in UI and committed to master.
> Documentation should have more information of LRU Stats
> ---
>
> Key: HBASE-8315
> URL: https://issues.apache.org/jira/browse/HBASE-8315
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 0.95.0
>Reporter: Daisuke Kobayashi
>Assignee: Misty Stanley-Jones
> Labels: documentation
> Fix For: 0.99.0
>
> Attachments: 8351.txt, Screen Shot 2014-06-13 at 10.59.05 AM.png
>
>
> Unfortunately, there's no documentation to explain the meaning of each LRU
> Stats in the regionserver logs. So this is for creating a new paragraph
> regarding this. My current idea is below, but it's a little bit difficult to
> explain the difference between 'cachingAccesses' and 'accesses' from an
> administrator or a user views.
> Could you guys help to improve the content?
> {noformat}
> total: The current memory size of the cache in use.
> free: The total free memory currently available to store more cache entries.
> max: Maximum allowed memory size of the cache.
> blocks: Caches store blocks of data; this number is the current # of blocks
> stored, which use up the "total" memory space.
> accesses: The total number of times the cache was accessed, regardless of
> result.
> hits: The total number of times the cache was accessed and the result was a
> successful hit (presence of looked up element in cache is a hit).
> hitRatio: The current percentage for "hits / accesses".
>
> Unclear:
> cachingAccesses: cachingHits + The number of getBlock requests that were
> cache misses, but only from requests that were set to use the block cache.
> cachingHits: The number of getBlock requests that were cache hits, but only
> from requests that were set to use the block cache. This is because all reads
> =
> cachingHitsRatio: The current percentage for "cachintHits / cachingAccesses"
> evictions: The total number of times an eviction has occurred (based on the
> use of the LRU algorithm)
> evicted: The total number of blocks that have been evicted (based on the use
> of the LRU algorithm)
> evictedPerRun: The total number of blocks that have been evicted overall /
> The number of times an eviction has occurred overall
> {noformat}
> And also, where should we add this paragraph in the documentation?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-8484) Adding diagnostic messages when the RegionServer hists an IOException in RegionServerReport
[ https://issues.apache.org/jira/browse/HBASE-8484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manukranth Kolloju resolved HBASE-8484. --- Resolution: Fixed > Adding diagnostic messages when the RegionServer hists an IOException in > RegionServerReport > --- > > Key: HBASE-8484 > URL: https://issues.apache.org/jira/browse/HBASE-8484 > Project: HBase > Issue Type: Task > Components: regionserver >Affects Versions: 0.89-fb >Reporter: Manukranth Kolloju >Priority: Trivial > Fix For: 0.89-fb > > > This diff adds instrumentation in the case when we catch IOException in > regionserver report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11348) Make frequency and sleep times of chaos monkeys configurable
Vandana Ayyalasomayajula created HBASE-11348: Summary: Make frequency and sleep times of chaos monkeys configurable Key: HBASE-11348 URL: https://issues.apache.org/jira/browse/HBASE-11348 Project: HBase Issue Type: Improvement Affects Versions: 0.98.3 Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor Currently the chaos monkeys used in the integration tests, run with a fixed configuration. It would be useful to have the frequency, sleep times to be configurable. That would help controlling the chaos the monkeys are intended to create. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11349) [Thrift] support authentication/impersonation
Jimmy Xiang created HBASE-11349: --- Summary: [Thrift] support authentication/impersonation Key: HBASE-11349 URL: https://issues.apache.org/jira/browse/HBASE-11349 Project: HBase Issue Type: Improvement Components: security, Thrift Reporter: Jimmy Xiang Assignee: Jimmy Xiang Thrift server can access HBase as a fixed authenticated user. However, we don't authenticate thrift clients. It will be great if the thrift server can authenticate clients, and support impersonation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11350) [PE] Allow random value size
stack created HBASE-11350: - Summary: [PE] Allow random value size Key: HBASE-11350 URL: https://issues.apache.org/jira/browse/HBASE-11350 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack Assignee: stack Allow PE to write random value sizes. Helpful mimic'ing 'real' sizings. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11351) Experiments with OffHeap memstores
Lars Hofhansl created HBASE-11351: - Summary: Experiments with OffHeap memstores Key: HBASE-11351 URL: https://issues.apache.org/jira/browse/HBASE-11351 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl I have been experimenting with offheap memstores using an offheap BTree implementation of ConcurrentMap provided by MapDB (www.mapdb.org) So far I hooked this up in an experimental way with inefficient serialization (in the sense that due to our special usage of the set api we need to back the set by a map, which in the off heap case means we need to serialize and store the value twice. It is used as key and value). But it works. It's quite a bit slower and due to need to deserialize upon scan it produces *more* short term garbage. Let's use this issue to discuss. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11352) When HMaster starts up it deletes the tmp snapshot directory, if you are exporting a snapshot at that time the job will fail
churro morales created HBASE-11352: -- Summary: When HMaster starts up it deletes the tmp snapshot directory, if you are exporting a snapshot at that time the job will fail Key: HBASE-11352 URL: https://issues.apache.org/jira/browse/HBASE-11352 Project: HBase Issue Type: Bug Affects Versions: 0.94.19 Reporter: churro morales We are exporting a very large table. The export snapshot job takes 7+ days to complete. During that time we had to bounce HMaster. When HMaster initializes, it initializes the SnapshotManager which subsequently deletes the .tmp directory. If this happens while the ExportSnapshot job is running the reference files get removed and the job fails. Maybe we could put some sort of token such that when this job is running HMaster wont reset the tmp directory. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Timestamp resolution
I'm saying that you do not always have to. You might have an external timestamp
oracle that does this for you and all I want is use that high resolution time
in HBase.
Unix has at least an API to get it:
int gettimeofday(struct timeval *tv, struct timezone *tz);
struct timeval {
time_t tv_sec; /* seconds */
suseconds_t tv_usec; /* microseconds */
};
Also if/when we figure out how to get this in Java we better have a place to
store it.
Maybe my requirement is too specific...?
-- Lars
From: Michael Segel
To: dev@hbase.apache.org
Sent: Thursday, June 12, 2014 11:06 AM
Subject: Re: Timestamp resolution
Silly question.
How do you get time in microseconds?
On Jun 12, 2014, at 2:56 PM, Andrew Purtell wrote:
> Thank you for the clarification, this is what threw me (the initial mail):
>
>> We do have 8 bytes for the TS, though. Not enough to store nanosecs (that
> would only cover 2^63/10^9/3600/24/365.24 = 292.279 years), but enough for
> microseconds (292279 years).
> Should we just store he TS is microseconds? We could do that right now (and
> just keep the ms resolution for now - i.e. the us part would always be 0
> for now).
>
>
> This says we might want to store a timestamp representation that can handle
> microsecond resolution. My next step was to wonder about the availability
> and practicality of microsecond resolution clocks. I don't take Michael's
> position of "don't".
>
>
>
> On Wed, Jun 11, 2014 at 10:39 PM, lars hofhansl wrote:
>
>> The issues you cite are all orthogonal. We have client/RS time now, we
>> have clock skew now, that is completely independent from the time
>> resolution.
>>
>>
>> I explained the need I saw for this before. Lemme include:
>>
>> On Fri, May 23, 2014 at 06:16PM, lars hofhansl wrote:
>>> The specific discussion here was a transaction engine doing snapshot
>>> isolation using the HBase timestamps, but still be close to wall clock
>> time
>>> as much as possible.
>>> In that scenario, with ms resolution you can only do 1000
>> transactions/sec,
>>> and so you need to turn the timestamp into something that is not wall
>> clock
>>> time as HBase understands it (and hence TTL, etc, will no longer work, as
>>> well as any other tools you've written that use the HBase timestamp).
>>> 1m transactions/sec are good enough (for now, I envision in a few years
>>> we'll be sitting here wondering how we could ever think that 1m
>>> transaction/sec are sufficient) :)
>>>
>>
>>
>> The point is: Even if you had timestamp oracle (that can resolve ms and
>> fill inside ms resolution with a counter), there'd be no way to use this as
>> the HBase timestamp while being close to wall clock (so that TTL, etc,
>> still works).
>> So specifically I was not advocating an automatic higher time resolution
>> (as far as I know that cannot be done reliably in Java across
>> multiple cores). I was advocating allowing clients with access to a
>> (perhaps, but not necessarily single threaded) timestamp oracle to store
>> those timestamps and still make use of all HBase optimization (filtering
>> HFiles, TTL, etc).
>>
>>
>> -- Lars
>>
>>
>>
>>
>> From: Michael Segel
>> To: dev@hbase.apache.org
>> Cc: lars hofhansl
>> Sent: Wednesday, June 11, 2014 2:03 PM
>> Subject: Re: Timestamp resolution
>>
>>
>> Weirdly enough I find that I have to agree with Andrew.
>>
>> First, how do you get time in units smaller than a ms?
>> Second clock skew becomes an issue.
>> Third, which clock are you using? The client machine? The RS? And then how
>> do you synchronize each of the RS to be within a ms of each other?
>> Correct me if I’m wrong but NTP doesn’t give that close of a sync.
>>
>> Sorry, but really, not a good idea.
>>
>> If you want this… you can store the temporal data as a column.
>>
>> Time really is relative.
>>
>>
>> On May 25, 2014, at 12:53 AM, Stack wrote:
>>
>>> On Fri, May 23, 2014 at 5:27 PM, lars hofhansl wrote:
>>>
We have discussed this in the past. It just came up again during an
internal discussion.
Currently we simply store a Java timestamp (millisec since epoch), i.e.
>> we
have ms resolution.
We do have 8 bytes for the TS, though. Not enough to store nanosecs
>> (that
would only cover 2^63/10^9/3600/24/365.24 = 292.279 years), but enough
>> for
microseconds (292279 years).
Should we just store he TS is microseconds? We could do that right now
(and just keep the ms resolution for now - i.e. the us part would
>> always be
0 for now).
Existing data must be in ms of course, so we'd grandfather that in, but
new tables could store by default in us.
We'd need to make this configurable both the column family level and
client level, so clients could still opt to see data in ms.
Comments? Too much to bite off?
-- Lars
>>> I'm a fan. As Enis
Re: Timestamp resolution
That's a fair point. I also would like to be able to make use of HBase timestamp based optimizations, such as filtering HFiles, etc. Maybe that still works as HBase should not make any assumption about the actual time here. -- Lars From: Jeffrey Zhong To: dev@hbase.apache.org Sent: Thursday, June 12, 2014 3:01 PM Subject: Re: Timestamp resolution In situations such as client "timestamp" is used to support transactions, it's not bad to disable server side TTL because TTL clean up doesn't care about transaction semantics. Therefore, it likely breaks data integrity across multiple tables or regions. It's still a good use case to support TTL on "client provided timestamp". I think we can provide a plug-in so that during compaction we allow user to interpret their own timestamp to decide if to GC old data. Thanks, -Jeffrey On 6/11/14 10:39 PM, "lars hofhansl" wrote: >The issues you cite are all orthogonal. We have client/RS time now, we >have clock skew now, that is completely independent from the time >resolution. > > >I explained the need I saw for this before. Lemme include: > >On Fri, May 23, 2014 at 06:16PM, lars hofhansl wrote: >> The specific discussion here was a transaction engine doing snapshot >> isolation using the HBase timestamps, but still be close to wall clock >>time >> as much as possible. >> In that scenario, with ms resolution you can only do 1000 >>transactions/sec, >> and so you need to turn the timestamp into something that is not wall >>clock >> time as HBase understands it (and hence TTL, etc, will no longer work, >>as >> well as any other tools you've written that use the HBase timestamp). >> 1m transactions/sec are good enough (for now, I envision in a few years >> we'll be sitting here wondering how we could ever think that 1m >> transaction/sec are sufficient) :) >> > > >The point is: Even if you had timestamp oracle (that can resolve ms and >fill inside ms resolution with a counter), there'd be no way to use this >as the HBase timestamp while being close to wall clock (so that TTL, etc, >still works). >So specifically I was not advocating an automatic higher time resolution >(as far as I know that cannot be done reliably in Java across >multiple cores). I was advocating allowing clients with access to a >(perhaps, but not necessarily single threaded) timestamp oracle to store >those timestamps and still make use of all HBase optimization (filtering >HFiles, TTL, etc). > > >-- Lars > > > > > From: Michael Segel >To: dev@hbase.apache.org >Cc: lars hofhansl >Sent: Wednesday, June 11, 2014 2:03 PM >Subject: Re: Timestamp resolution > > >Weirdly enough I find that I have to agree with Andrew. > >First, how do you get time in units smaller than a ms? >Second clock skew becomes an issue. >Third, which clock are you using? The client machine? The RS? And then >how do you synchronize each of the RS to be within a ms of each other? >Correct me if I¹m wrong but NTP doesn¹t give that close of a sync. > >Sorry, but really, not a good idea. > >If you want thisŠ you can store the temporal data as a column. > >Time really is relative. > > >On May 25, 2014, at 12:53 AM, Stack wrote: > >> On Fri, May 23, 2014 at 5:27 PM, lars hofhansl wrote: >> >>> We have discussed this in the past. It just came up again during an >>> internal discussion. >>> Currently we simply store a Java timestamp (millisec since epoch), >>>i.e. we >>> have ms resolution. >>> >>> We do have 8 bytes for the TS, though. Not enough to store nanosecs >>>(that >>> would only cover 2^63/10^9/3600/24/365.24 = 292.279 years), but enough >>>for >>> microseconds (292279 years). >>> Should we just store he TS is microseconds? We could do that right now >>> (and just keep the ms resolution for now - i.e. the us part would >>>always be >>> 0 for now). >>> Existing data must be in ms of course, so we'd grandfather that in, but >>> new tables could store by default in us. >>> >>> We'd need to make this configurable both the column family level and >>> client level, so clients could still opt to see data in ms. >>> >>> Comments? Too much to bite off? >>> >>> -- Lars >>> >>> >> I'm a fan. As Enis cites, HBASE-8927 has good discussion. No >> configuration I'd say. Just move to the new regime (though I suppose we >> should let you turn it off). >> >> I think it was Liu Shaohui (IIRC) who made a suggestion that had us put >> together ms and nanos under a synchronized block stamping the ts on >>Cells >> (left-shift the currentTimeMillis and fill in the bottom bytes with as >>much >> of the nanos as fits; i.e. your micros). Rather than nanos/micros, we >> could use a counter instead if a Cell arrives in the same ms. Would be >> costly having all ops go via one code block to get 'time' across cores >>and >> handlers. >> >> St.Ack -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it i
Time for 0.94.21
Only 6 fixes so far (a few in the pipeline). 0.94 is clearly stabilizing after 20 releases - or maybe interest has just shifted to 0.96 and later. In any case, thinking about rolling an RC next week. Let me know about any issue you'd like to get in. Thanks. -- Lars
