[jira] [Assigned] (HBASE-7077) Test for: CheckAndPut should properly read MVCC
[ https://issues.apache.org/jira/browse/HBASE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-7077: - Assignee: ramkrishna.s.vasudevan > Test for: CheckAndPut should properly read MVCC > --- > > Key: HBASE-7077 > URL: https://issues.apache.org/jira/browse/HBASE-7077 > Project: HBase > Issue Type: Sub-task >Reporter: Gregory Chanan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-7071.patch, HBASE-7071_testcasewithassert.patch > > > checkAndPut should integrate with MVCC, similar to how HBASE-4583 fixed > appends and increments. > Also need a test, here's one we could use (originally proposed in HBASE-7051): > The current value of some cell is 10. > I issue two concurrent requests: > A) a check and put where check value = 10, put value = 11 > B) a put where put value = 50 > The only result at the end of these operations that seems reasonable to me is > the value of the cell being 50. If A occurred first (ACID wise), then our > values go 10->11->50. If B occurred first, then our values go 10->50 (and the > checkAndPut fails) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488516#comment-13488516 ] Hudson commented on HBASE-6852: --- Integrated in HBase-0.94 #562 (See [https://builds.apache.org/job/HBase-0.94/562/]) HBASE-6852 SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields (Cheng Hao and LarsH) - REAPPLY (Revision 1404464) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Assignee: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3 > > Attachments: 6852-0.94_2.patch, 6852-0.94.txt, metrics_hotspots.png, > onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[ https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488515#comment-13488515 ] Hudson commented on HBASE-6904: --- Integrated in HBase-0.94 #562 (See [https://builds.apache.org/job/HBase-0.94/562/]) HBASE-6904 In the HBase shell, an error is thrown that states replication-related znodes already exist (Revision 1404385) Result = FAILURE gchanan : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java > In the HBase shell, an error is thrown that states replication-related znodes > already exist > --- > > Key: HBASE-6904 > URL: https://issues.apache.org/jira/browse/HBASE-6904 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Affects Versions: 0.92.0, 0.92.1, 0.96.0 >Reporter: Aleksandr Shulman >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6904-94.patch, HBASE-6904.patch > > > On a replication-enabled cluster, querying the list_peers produces the error > lines shown below. It doesn't appear that anything is broken in terms of > functionality. > Stack trace: > hbase(main):001:0> list_peers > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/peers already exists and this is not a retry > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/rs already exists and this is not a retry > PEER ID CLUSTER KEY > 0 row(s) in 0.4650 seconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7051) CheckAndPut should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488513#comment-13488513
]
Hudson commented on HBASE-7051:
---
Integrated in HBase-0.94 #562 (See
[https://builds.apache.org/job/HBase-0.94/562/])
HBASE-7051 CheckAndPut should properly read MVCC (Revision 1404379)
Result = FAILURE
larsh :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> CheckAndPut should properly read MVCC
> -
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7051.txt
>
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7077) Test for: CheckAndPut should properly read MVCC
[ https://issues.apache.org/jira/browse/HBASE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488514#comment-13488514 ] Hudson commented on HBASE-7077: --- Integrated in HBase-0.94 #562 (See [https://builds.apache.org/job/HBase-0.94/562/]) HBASE-7077 Test for: CheckAndPut should properly read MVCC (Ram) (Revision 1404383) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHBase7051.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java > Test for: CheckAndPut should properly read MVCC > --- > > Key: HBASE-7077 > URL: https://issues.apache.org/jira/browse/HBASE-7077 > Project: HBase > Issue Type: Sub-task >Reporter: Gregory Chanan > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-7071.patch, HBASE-7071_testcasewithassert.patch > > > checkAndPut should integrate with MVCC, similar to how HBASE-4583 fixed > appends and increments. > Also need a test, here's one we could use (originally proposed in HBASE-7051): > The current value of some cell is 10. > I issue two concurrent requests: > A) a check and put where check value = 10, put value = 11 > B) a put where put value = 50 > The only result at the end of these operations that seems reasonable to me is > the value of the cell being 50. If A occurred first (ACID wise), then our > values go 10->11->50. If B occurred first, then our values go 10->50 (and the > checkAndPut fails) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488493#comment-13488493
]
Lars Hofhansl commented on HBASE-6317:
--
So should we commit this 0.92/0.94?
> Master clean start up and Partially enabled tables make region assignment
> inconsistent.
> ---
>
> Key: HBASE-6317
> URL: https://issues.apache.org/jira/browse/HBASE-6317
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: rajeshbabu
> Fix For: 0.92.3, 0.94.3, 0.96.0
>
> Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch,
> HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch,
> HBASE-6317_trunk_5.patch
>
>
> If we have a table in partially enabled state (ENABLING) then on HMaster
> restart we treat it as a clean cluster start up and do a bulk assign.
> Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
> leads to region assignment problems. Analysing more on this we found that we
> have better way to handle these scenarios.
> {code}
> if (false == checkIfRegionBelongsToDisabled(regionInfo)
> && false == checkIfRegionsBelongsToEnabling(regionInfo)) {
> synchronized (this.regions) {
> regions.put(regionInfo, regionLocation);
> addToServers(regionLocation, regionInfo);
> }
> {code}
> We dont add to regions map so that enable table handler can handle it. But
> as nothing is added to regions map we think it as a clean cluster start up.
> Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6391) Master restart when enabling table will lead to region assignned twice
[
https://issues.apache.org/jira/browse/HBASE-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6391:
-
Fix Version/s: (was: 0.94.3)
No update, unscheduling.
> Master restart when enabling table will lead to region assignned twice
> --
>
> Key: HBASE-6391
> URL: https://issues.apache.org/jira/browse/HBASE-6391
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
>
> The Scenario can be reproduce below.
> Enabling an table, some region is online on regionserver,some are still being
> processed.
> And restart the master.
> when master failover:
> // Region is being served and on an active server
> // add only if region not in disabled and enabling table
> if (false == checkIfRegionBelongsToDisabled(regionInfo)
> && false == checkIfRegionsBelongsToEnabling(regionInfo)) {
> regions.put(regionInfo, regionLocation);
> addToServers(regionLocation, regionInfo);
> }
> the opened region will not add to the Regions in master.
> and in the following recoverTableInEnablingState,the region will be assigned
> again.
> that will lead to the cluster inconsistent
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488488#comment-13488488 ] Lars Hofhansl commented on HBASE-6060: -- What's the feeling w.r.t. 0.92 and 0.94? > Regions's in OPENING state from failed regionservers takes a long time to > recover > - > > Key: HBASE-6060 > URL: https://issues.apache.org/jira/browse/HBASE-6060 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Reporter: Enis Soztutar >Assignee: rajeshbabu > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, > 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, > 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, > 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, > 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, > HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, > HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch, > trunk-6060.patch, trunk-6060_v2.patch > > > we have seen a pattern in tests, that the regions are stuck in OPENING state > for a very long time when the region server who is opening the region fails. > My understanding of the process: > > - master calls rs to open the region. If rs is offline, a new plan is > generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in > master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), > HMaster.assign() > - RegionServer, starts opening a region, changes the state in znode. But > that znode is not ephemeral. (see ZkAssign) > - Rs transitions zk node from OFFLINE to OPENING. See > OpenRegionHandler.process() > - rs then opens the region, and changes znode from OPENING to OPENED > - when rs is killed between OPENING and OPENED states, then zk shows OPENING > state, and the master just waits for rs to change the region state, but since > rs is down, that wont happen. > - There is a AssignmentManager.TimeoutMonitor, which does exactly guard > against these kind of conditions. It periodically checks (every 10 sec by > default) the regions in transition to see whether they timedout > (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, > which explains what you and I are seeing. > - ServerShutdownHandler in Master does not reassign regions in OPENING > state, although it handles other states. > Lowering that threshold from the configuration is one option, but still I > think we can do better. > Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7060) Region load balancing by table does not handle the case where a table's region count is lower than the number of the RS in the cluster
[
https://issues.apache.org/jira/browse/HBASE-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-7060:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Marking as fixed.
> Region load balancing by table does not handle the case where a table's
> region count is lower than the number of the RS in the cluster
> --
>
> Key: HBASE-7060
> URL: https://issues.apache.org/jira/browse/HBASE-7060
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.92.0
>Reporter: Tianying Chang
>Assignee: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7060-94.txt, 7060-94-v2.txt, 7060-test-tentative-94.txt,
> 7060-trunk.txt, HBASE-7060.patch
>
>
> When the table's region count is less than the count of region servers, the
> region balance algorithm will not move the region. For example, the cluster
> has 100 RS, the table has 50 regions sitting on one RS, they will not be
> moved to any of the other 99 RS.
> This is because the algorithm did not calculate the under-loaded RS
> correctly. This is how the algorithm works with the above example:
> avg-regions-per-RS=0.5
> min-RS-per-RS=0
> max-RS-per-RS=1
> when they calculate the under loaded RS, the code is as below. Since
> regionCount=0, which is always >=min, so it will always skip, therefore, no
> underloaded RS are found.
> Map underloadedServers = new HashMap Integer>();
> for (Map.Entry> server:
> serversByLoad.entrySet()) {
> int regionCount = server.getKey().getLoad();
> if (regionCount >= min) { break; }
> underloadedServers.put(server.getKey().getServerName(), min - regionCount);
> }
> Later the function returns since underloaded RS size is 0
> if (serverUnerloaded ==0) return regionsToReturn;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6389) Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments
[
https://issues.apache.org/jira/browse/HBASE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488483#comment-13488483
]
Lars Hofhansl commented on HBASE-6389:
--
I do not know this area well.
[~ram_krish] Wanna have a look too?
> Modify the conditions to ensure that Master waits for sufficient number of
> Region Servers before starting region assignments
>
>
> Key: HBASE-6389
> URL: https://issues.apache.org/jira/browse/HBASE-6389
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.94.0, 0.96.0
>Reporter: Aditya Kishore
>Assignee: Aditya Kishore
>Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: HBASE-6389_0.94.patch, HBASE-6389_trunk.patch,
> HBASE-6389_trunk.patch, HBASE-6389_trunk.patch, HBASE-6389_trunk_v2.patch,
> HBASE-6389_trunk_v2.patch, org.apache.hadoop.hbase.TestZooKeeper-output.txt,
> testReplication.jstack
>
>
> Continuing from HBASE-6375.
> It seems I was mistaken in my assumption that changing the value of
> "hbase.master.wait.on.regionservers.mintostart" to a sufficient number (from
> default of 1) can help prevent assignment of all regions to one (or a small
> number of) region server(s).
> While this was the case in 0.90.x and 0.92.x, the behavior has changed in
> 0.94.0 onwards to address HBASE-4993.
> From 0.94.0 onwards, Master will proceed immediately after the timeout has
> lapsed, even if "hbase.master.wait.on.regionservers.mintostart" has not
> reached.
> Reading the current conditions of waitForRegionServers() clarifies it
> {code:title=ServerManager.java (trunk rev:1360470)}
>
> 581 /**
> 582 * Wait for the region servers to report in.
> 583 * We will wait until one of this condition is met:
> 584 * - the master is stopped
> 585 * - the 'hbase.master.wait.on.regionservers.timeout' is reached
> 586 * - the 'hbase.master.wait.on.regionservers.maxtostart' number of
> 587 *region servers is reached
> 588 * - the 'hbase.master.wait.on.regionservers.mintostart' is reached
> AND
> 589 * there have been no new region server in for
> 590 * 'hbase.master.wait.on.regionservers.interval' time
> 591 *
> 592 * @throws InterruptedException
> 593 */
> 594 public void waitForRegionServers(MonitoredTask status)
> 595 throws InterruptedException {
>
>
> 612 while (
> 613 !this.master.isStopped() &&
> 614 slept < timeout &&
> 615 count < maxToStart &&
> 616 (lastCountChange+interval > now || count < minToStart)
> 617 ){
>
> {code}
> So with the current conditions, the wait will end as soon as timeout is
> reached even lesser number of RS have checked-in with the Master and the
> master will proceed with the region assignment among these RSes alone.
> As mentioned in
> -[HBASE-4993|https://issues.apache.org/jira/browse/HBASE-4993?focusedCommentId=13237196#comment-13237196]-,
> and I concur, this could have disastrous effect in large cluster especially
> now that MSLAB is turned on.
> To enforce the required quorum as specified by
> "hbase.master.wait.on.regionservers.mintostart" irrespective of timeout,
> these conditions need to be modified as following
> {code:title=ServerManager.java}
> ..
> /**
>* Wait for the region servers to report in.
>* We will wait until one of this condition is met:
>* - the master is stopped
>* - the 'hbase.master.wait.on.regionservers.maxtostart' number of
>*region servers is reached
>* - the 'hbase.master.wait.on.regionservers.mintostart' is reached AND
>* there have been no new region server in for
>* 'hbase.master.wait.on.regionservers.interval' time AND
>* the 'hbase.master.wait.on.regionservers.timeout' is reached
>*
>* @throws InterruptedException
>*/
> public void waitForRegionServers(MonitoredTask status)
> ..
> ..
> int minToStart = this.master.getConfiguration().
> getInt("hbase.master.wait.on.regionservers.mintostart", 1);
> int maxToStart = this.master.getConfiguration().
> getInt("hbase.master.wait.on.regionservers.maxtostart",
> Integer.MAX_VALUE);
> if (maxToStart < minToStart) {
> maxToStart = minToStart;
> }
> ..
> ..
> while (
> !this.master.isStopped() &&
> count < maxToStart &&
> (lastCountChange+interval > now || timeout > slept || count <
> minToStart)
> ){
> ..
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/
[jira] [Commented] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
[ https://issues.apache.org/jira/browse/HBASE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488482#comment-13488482 ] stack commented on HBASE-6925: -- Good blast from the past Lars. Would the chunking at 8k undo some of the benefit of 64k 'ing it? > Change socket write size from 8K to 64K for HBaseServer > --- > > Key: HBASE-6925 > URL: https://issues.apache.org/jira/browse/HBASE-6925 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6925.patch > > > Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT > from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488479#comment-13488479 ] Lars Hofhansl commented on HBASE-5898: -- Yeah, offheap caching for diskblocks always seemed a bit "interesting" as the blocks all have the same size the GC should be able to deal with the churn nicely. > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488478#comment-13488478 ] stack commented on HBASE-5898: -- [~eclark] Elliott, you get any more data on the hung up case (Elliott has been trying to repo it so can get more data on slow wait/notifying) > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488477#comment-13488477 ] stack commented on HBASE-5898: -- The offheap cache is being used by no one -- its unusable in current form -- and doubtful anyone would ever use it or if it even makes sense to use it as currently implemented instead of just having bigger block cache w/ decent GC tuning + file system cache instead. I'd be fine for ripping it out until someone wanted to take it up and finish it off. > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
[ https://issues.apache.org/jira/browse/HBASE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488476#comment-13488476 ] Lars Hofhansl commented on HBASE-6925: -- Is this at all related to HBASE-4956? Do these two need to be coordinated? > Change socket write size from 8K to 64K for HBaseServer > --- > > Key: HBASE-6925 > URL: https://issues.apache.org/jira/browse/HBASE-6925 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6925.patch > > > Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT > from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
[ https://issues.apache.org/jira/browse/HBASE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488475#comment-13488475 ] stack commented on HBASE-6925: -- +1 on brother Karthiks' word. > Change socket write size from 8K to 64K for HBaseServer > --- > > Key: HBASE-6925 > URL: https://issues.apache.org/jira/browse/HBASE-6925 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6925.patch > > > Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT > from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488474#comment-13488474 ] stack commented on HBASE-7055: -- I added a review Sergey. Good stuff. > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, > HBASE-6371-v3-refactor-only-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6874) Implement prefetching for scanners
[ https://issues.apache.org/jira/browse/HBASE-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488473#comment-13488473 ] Lars Hofhansl commented on HBASE-6874: -- Yeah, it's tricky to do that at the Scanner level. In our case we have N ClientScanners. We break up the scan into chunks and for each chunk we use a separate ClientScanner (in a nutshell). We then sort the chunks (only the chunks not all the KVs) at the client based on the startkey for that chunk. Some of our usecases do relatively large scans (hundreds of millions of rows), and we want to engage many cores and spindles at the RegionServers in parallel (we control the level of parallelism we want by the chunking)... This is for online analytics over preaggregated data. It's quite possible that our use case is too special to fit into any kind of generalized scheme. > Implement prefetching for scanners > -- > > Key: HBASE-6874 > URL: https://issues.apache.org/jira/browse/HBASE-6874 > Project: HBase > Issue Type: Sub-task >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan > > I did some quick experiments by scanning data that should be completely in > memory and found that adding pre-fetching increases the throughput by about > 50% from 26MB/s to 39MB/s. > The idea is to perform the next in a background thread, and keep the result > ready. When the scanner's next comes in, return the pre-computed result and > issue another background read. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488470#comment-13488470 ] Lars Hofhansl commented on HBASE-5898: -- Offtopic: Is the offheap cache even working currently? If not I'd be in favor of ripping that code out. > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.
[ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488468#comment-13488468 ] Lars Hofhansl commented on HBASE-6796: -- I am going to commit tomorrow unless I hear objections. > Backport HBASE-5547, Don't delete HFiles in backup mode. > > > Key: HBASE-6796 > URL: https://issues.apache.org/jira/browse/HBASE-6796 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Jesse Yates > Fix For: 0.94.3 > > Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch > > > See HBASE-5547 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
[ https://issues.apache.org/jira/browse/HBASE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488466#comment-13488466 ] Lars Hofhansl commented on HBASE-6925: -- Going to commit to 0.94 and 0.96 tomorrow. > Change socket write size from 8K to 64K for HBaseServer > --- > > Key: HBASE-6925 > URL: https://issues.apache.org/jira/browse/HBASE-6925 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6925.patch > > > Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT > from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6919) Remove unnecessary cast from Bytes.readVLong
[
https://issues.apache.org/jira/browse/HBASE-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6919:
-
Fix Version/s: (was: 0.94.3)
0.94.4
> Remove unnecessary cast from Bytes.readVLong
>
>
> Key: HBASE-6919
> URL: https://issues.apache.org/jira/browse/HBASE-6919
> Project: HBase
> Issue Type: Bug
>Reporter: James Taylor
>Priority: Minor
> Fix For: 0.96.0, 0.94.4
>
>
> Remove the throws IOException so that caller doesn't have to catch and ignore.
> {code}
> public static long readVLong(final byte [] buffer, final int offset)
> throws IOException
> {code}
> Also, add
> {code}
> public static int readVInt(final byte [] buffer, final int offset)
> throws IOException {
> return (int)readVLong(buffer,offset);
> }
> {code}
> and these are useful too:
> {code}
> /**
> * Put long as variable length encoded number at the offset in
> * the result byte array.
> * @param vint Integer to make a vint of.
> * @param result buffer to put vint into
> * @return Vint length in bytes of vint
> */
> public static int vintToBytes(byte[] result, int offset, final long vint)
> {
> long i = vint;
> if (i >= -112 && i <= 127) {
> result[offset] = (byte) i;
> return 1;
> }
> int len = -112;
> if (i < 0) {
> i ^= -1L; // take one's complement'
> len = -120;
> }
> long tmp = i;
> while (tmp != 0) {
> tmp = tmp >> 8;
> len--;
> }
> result[offset++] = (byte) len;
> len = (len < -120) ? -(len + 120) : -(len + 112);
> for (int idx = len; idx != 0; idx--) {
> int shiftbits = (idx - 1) * 8;
> long mask = 0xFFL << shiftbits;
> result[offset++] = (byte)((i & mask) >> shiftbits);
> }
> return len + 1;
> }
> /**
> * Decode a vint from the buffer pointed at to by ptr and
> * increment the offset of the ptr by the length of the
> * vint.
> * @param ptr a pointer to a byte array buffer
> * @return the decoded vint value as an int
> */
> public static int vintFromBytes(ImmutableBytesWritable ptr) {
> return (int) vlongFromBytes(ptr);
> }
>
> /**
> * Decode a vint from the buffer pointed at to by ptr and
> * increment the offset of the ptr by the length of the
> * vint.
> * @param ptr a pointer to a byte array buffer
> * @return the decoded vint value as a long
> */
> public static long vlongFromBytes(ImmutableBytesWritable ptr) {
> final byte [] buffer = ptr.get();
> final int offset = ptr.getOffset();
> byte firstByte = buffer[offset];
> int len = WritableUtils.decodeVIntSize(firstByte);
> if (len == 1) {
> ptr.set(buffer, offset+1, ptr.getLength());
> return firstByte;
> }
> long i = 0;
> for (int idx = 0; idx < len-1; idx++) {
> byte b = buffer[offset + 1 + idx];
> i = i << 8;
> i = i | (b & 0xFF);
> }
> ptr.set(buffer, offset+len, ptr.getLength());
> return (WritableUtils.isNegativeVInt(firstByte) ? ~i : i);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-6852. -- Resolution: Fixed I ran all the tests locally as well. All pass now. Committed to 0.94. Thanks for the patch and the fix Cheng! > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Assignee: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3 > > Attachments: 6852-0.94_2.patch, 6852-0.94.txt, metrics_hotspots.png, > onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6874) Implement prefetching for scanners
[ https://issues.apache.org/jira/browse/HBASE-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488463#comment-13488463 ] Karthik Ranganathan commented on HBASE-6874: Thought about the N scanners, its a complicated change - you would have to change the entire scan protocol. Each of the next calls in scans are not numbered, and so you could go out of whack if prefetching N (and throw in exceptions). There is also the basic issue right now that scans do retries which is wrong. Also, reasoning about it another way, if your in memory scan throughput is > the time to read from disk, you're probably good. I found that there are other unrelated bottlenecks preventing this from being the case. Of course, if the filtering is very heavy then this will breakdown... you probably want to implement prefetching based on the num filtered rows, which should not be too hard. I have a patch I have tested with, but its waiting on HBASE-6770 - that is going to refactor scans quite a bit. Will put a patch out once that is done. > Implement prefetching for scanners > -- > > Key: HBASE-6874 > URL: https://issues.apache.org/jira/browse/HBASE-6874 > Project: HBase > Issue Type: Sub-task >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan > > I did some quick experiments by scanning data that should be completely in > memory and found that adding pre-fetching increases the throughput by about > 50% from 26MB/s to 39MB/s. > The idea is to perform the next in a background thread, and keep the result > ready. When the scanner's next comes in, return the pre-computed result and > issue another background read. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488460#comment-13488460 ] stack commented on HBASE-7055: -- [~sershe] On documentation, a section in the book would be cool but failing that, a release note should describe what the feature does, why you would use it and its configuration params (I'd think). It can be terse and expanded later in the book but w/o such a thing, the code is often committed and just forgotten (I looked at original JIRA and it has little by way of description. I fixed the bad link in the README.txt. Thanks for pointing it out). Refactor, a good idea I think, should be done in a new issue. Can cite the posted rb (I'll try help out w/ a review now). I like the scheme you fellas have for putting policy in table (rather than in a reloadable config. that is for compaction only) If the class names are confusing, lets fix them in refactor (same as 89fb branch is not good enough reason preserving names that confound) > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, > HBASE-6371-v3-refactor-only-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
[ https://issues.apache.org/jira/browse/HBASE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488459#comment-13488459 ] Karthik Ranganathan commented on HBASE-6925: Missed Stack's question - this complex change alone gave a 25% best case improvement in scan throughput. > Change socket write size from 8K to 64K for HBaseServer > --- > > Key: HBASE-6925 > URL: https://issues.apache.org/jira/browse/HBASE-6925 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6925.patch > > > Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT > from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Hao updated HBASE-6852: - Attachment: 6852-0.94_2.patch Please take the 6852-0.94_2.patch Found a small bug while updating the "ALL_CATEGORY metrics" in the function "SchemaMetrics.incrNumericMetric(BlockCategory blockCategory, boolean isCompaction, BlockMetricType metricType, long amount)" And I also add the "flushMetrics()" in the function "SchemaMetrics.getMetricsSnapshot()" Now it passed the unit test in my local. Thank you. > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Assignee: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3 > > Attachments: 6852-0.94_2.patch, 6852-0.94.txt, metrics_hotspots.png, > onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
[ https://issues.apache.org/jira/browse/HBASE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488455#comment-13488455 ] Lars Hofhansl commented on HBASE-6925: -- +1 on committing to 0.94 and 0.96. > Change socket write size from 8K to 64K for HBaseServer > --- > > Key: HBASE-6925 > URL: https://issues.apache.org/jira/browse/HBASE-6925 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6925.patch > > > Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT > from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
[ https://issues.apache.org/jira/browse/HBASE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6925: - Fix Version/s: 0.96.0 0.94.3 I agree. This should go into 0.94 as well. > Change socket write size from 8K to 64K for HBaseServer > --- > > Key: HBASE-6925 > URL: https://issues.apache.org/jira/browse/HBASE-6925 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6925.patch > > > Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT > from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6874) Implement prefetching for scanners
[ https://issues.apache.org/jira/browse/HBASE-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488448#comment-13488448 ] Lars Hofhansl commented on HBASE-6874: -- [~karthik.ranga] Do you have a patch. We were just discussing something similar here and I was about to open a similar issue before I found this one. This is even more useful with scanner caching. One could even go a step further and parallelize the prefetching into N threads (useful if the results are heavily prefiltered at the server). We do our own parallel scanner fetching (not necessarily on region or buffer boundaries), but it would be nice if that could be generalized and be part of HBase. > Implement prefetching for scanners > -- > > Key: HBASE-6874 > URL: https://issues.apache.org/jira/browse/HBASE-6874 > Project: HBase > Issue Type: Sub-task >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan > > I did some quick experiments by scanning data that should be completely in > memory and found that adding pre-fetching increases the throughput by about > 50% from 26MB/s to 39MB/s. > The idea is to perform the next in a background thread, and keep the result > ready. When the scanner's next comes in, return the pre-computed result and > issue another background read. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7083) SSH#fixupDaughter should force re-assign missing daughter
[ https://issues.apache.org/jira/browse/HBASE-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488445#comment-13488445 ] stack commented on HBASE-7083: -- Patch looks fine Jimmy but why do we have to force the reassign? Why does it not work w/o forcing it? > SSH#fixupDaughter should force re-assign missing daughter > - > > Key: HBASE-7083 > URL: https://issues.apache.org/jira/browse/HBASE-7083 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.96.0 > > Attachments: trunk-7083.patch > > > In looking into flaky test > TestSplitTransactionOnCluster#testShutdownSimpleFixup, I found out that a > missing daughter is not assigned by SSH properly. It could be open on the > dead server. We need to force re-assign it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7083) SSH#fixupDaughter should force re-assign missing daughter
[ https://issues.apache.org/jira/browse/HBASE-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488440#comment-13488440 ] Ted Yu commented on HBASE-7083: --- Patch looks good. Do you mind submitting for QA run again ? > SSH#fixupDaughter should force re-assign missing daughter > - > > Key: HBASE-7083 > URL: https://issues.apache.org/jira/browse/HBASE-7083 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.96.0 > > Attachments: trunk-7083.patch > > > In looking into flaky test > TestSplitTransactionOnCluster#testShutdownSimpleFixup, I found out that a > missing daughter is not assigned by SSH properly. It could be open on the > dead server. We need to force re-assign it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488434#comment-13488434 ] Phabricator commented on HBASE-7031: pritamdamania has commented on the revision "[jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat". I'm not sure about the duplicate use of mapred and mapreduce .. REVISION DETAIL https://reviews.facebook.net/D6129 BRANCH arcpatch-D6129 To: Kannan, Karthik, JIRA, mbautin, aaiyer, Liyin, pritamdamania Cc: tjackson > Support startRow/endRow in TableInputFormat > --- > > Key: HBASE-7031 > URL: https://issues.apache.org/jira/browse/HBASE-7031 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Priority: Minor > Attachments: D6129.3.patch, D6129.4.patch > > > We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as > opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop > Streaming integration. We need to add startRow/endRow support to > TableInputFormat due to a product requirement. However, the two > TableInputFormat implementations have diverged over time. We need to make > mapred.TableInputFormat reuse some of the newer code in > mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488427#comment-13488427 ] Phabricator commented on HBASE-7031: Liyin has accepted the revision "[jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat". The diff looks good to me. BTW, why does hadoop streaming still depend on the mapred package ? If we can deprecated the mapred package, it helps to remove the duplicated code path. REVISION DETAIL https://reviews.facebook.net/D6129 BRANCH arcpatch-D6129 To: Kannan, Karthik, JIRA, mbautin, aaiyer, Liyin, pritamdamania Cc: tjackson > Support startRow/endRow in TableInputFormat > --- > > Key: HBASE-7031 > URL: https://issues.apache.org/jira/browse/HBASE-7031 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Priority: Minor > Attachments: D6129.3.patch, D6129.4.patch > > > We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as > opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop > Streaming integration. We need to add startRow/endRow support to > TableInputFormat due to a product requirement. However, the two > TableInputFormat implementations have diverged over time. We need to make > mapred.TableInputFormat reuse some of the newer code in > mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488412#comment-13488412 ] Phabricator commented on HBASE-7031: pritamdamania has added reviewers to the revision "[jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat". Added Reviewers: aaiyer, Liyin Could someone take a look at this diff ? REVISION DETAIL https://reviews.facebook.net/D6129 To: Kannan, Karthik, JIRA, mbautin, aaiyer, Liyin, pritamdamania Cc: tjackson > Support startRow/endRow in TableInputFormat > --- > > Key: HBASE-7031 > URL: https://issues.apache.org/jira/browse/HBASE-7031 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Priority: Minor > Attachments: D6129.3.patch, D6129.4.patch > > > We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as > opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop > Streaming integration. We need to add startRow/endRow support to > TableInputFormat due to a product requirement. However, the two > TableInputFormat implementations have diverged over time. We need to make > mapred.TableInputFormat reuse some of the newer code in > mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488408#comment-13488408 ] Sergey Shelukhin commented on HBASE-7055: - Created review at https://reviews.apache.org/r/7823/ to make it easier to give feedback. > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, > HBASE-6371-v3-refactor-only-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488403#comment-13488403 ] Enis Soztutar commented on HBASE-7055: -- Whatever we name it, it should be consistent. The patch uses CompactionPolicy, CompactionManager, TierCompactionManager and Tier Compaction Policy interchangeably. > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, > HBASE-6371-v3-refactor-only-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6793) Make hbase-examples module
[ https://issues.apache.org/jira/browse/HBASE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-6793: Attachment: HBASE-6793-v5-squashed.patch whitespace (from CR feedback) > Make hbase-examples module > -- > > Key: HBASE-6793 > URL: https://issues.apache.org/jira/browse/HBASE-6793 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Enis Soztutar >Assignee: Sergey Shelukhin > Labels: noob > Attachments: HBASE-6793.patch, HBASE-6793-v2.patch, > HBASE-6793-v3-thrift-0.9.0.patch, HBASE-6793-v4.1-squashed.patch, > HBASE-6793-v5-squashed.patch > > > There are some examples under /examples/, which are not compiled as a part of > the build. We can move them to an hbase-examples module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7042) Master Coprocessor Endpoint
[
https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488394#comment-13488394
]
Francis Liu commented on HBASE-7042:
{quote}
Can we use the same Exec and ExecResponse classes for both master and RS
endpoint invocations? (I.e. get rid of MasterExec and MasterExecResult?)
{quote}
It's not possible since the region classes contains relations to
regions/row/etc.
{quote}
The patch handles system coprocessors, those registered by Configuration, but
what about table coprocessors? Table coprocessors are loaded as dictated by
table attributes. Should the master load those when creating or updating table
schema and discovers that coprocessor load directives now exist in the schema?
We wouldn't need to load a table coprocessor into the master if the table is
disabled, but once the table is enabled, then we would?
{quote}
Do we really need a table specific MasterEndpoint? For Region's it was really
useful since it'd enable you to do distributed computation across regions apart
from being able to extend Region/RegionServer functionality. In the master's
case, most use cases would be to extend the master's functionality?
> Master Coprocessor Endpoint
> ---
>
> Key: HBASE-7042
> URL: https://issues.apache.org/jira/browse/HBASE-7042
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Liu
>Assignee: Francis Liu
> Fix For: 0.96.0
>
> Attachments: HBASE-7042_94.patch
>
>
> Having support for a master coprocessor endpoint would enable developers to
> easily extended HMaster functionality/features. As is the case for region
> server grouping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488382#comment-13488382 ] Sergey Shelukhin commented on HBASE-7055: - the refactor patch attached > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, > HBASE-6371-v3-refactor-only-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7042) Master Coprocessor Endpoint
[ https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488384#comment-13488384 ] Francis Liu commented on HBASE-7042: Gary, Yep, I'll be working on a trunk patch as well. I did 0.94 first as it's what we'll be deploying to production. This change shouldn't break binary backward compatibity for 0.94? > Master Coprocessor Endpoint > --- > > Key: HBASE-7042 > URL: https://issues.apache.org/jira/browse/HBASE-7042 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.96.0 > > Attachments: HBASE-7042_94.patch > > > Having support for a master coprocessor endpoint would enable developers to > easily extended HMaster functionality/features. As is the case for region > server grouping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7055: Attachment: HBASE-6371-v3-refactor-only-squashed.patch > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, > HBASE-6371-v3-refactor-only-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7083) SSH#fixupDaughter should force re-assign missing daughter
[ https://issues.apache.org/jira/browse/HBASE-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7083: --- Fix Version/s: 0.96.0 Status: Patch Available (was: Open) > SSH#fixupDaughter should force re-assign missing daughter > - > > Key: HBASE-7083 > URL: https://issues.apache.org/jira/browse/HBASE-7083 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.96.0 > > Attachments: trunk-7083.patch > > > In looking into flaky test > TestSplitTransactionOnCluster#testShutdownSimpleFixup, I found out that a > missing daughter is not assigned by SSH properly. It could be open on the > dead server. We need to force re-assign it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7083) SSH#fixupDaughter should force re-assign missing daughter
[ https://issues.apache.org/jira/browse/HBASE-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7083: --- Attachment: trunk-7083.patch > SSH#fixupDaughter should force re-assign missing daughter > - > > Key: HBASE-7083 > URL: https://issues.apache.org/jira/browse/HBASE-7083 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.96.0 > > Attachments: trunk-7083.patch > > > In looking into flaky test > TestSplitTransactionOnCluster#testShutdownSimpleFixup, I found out that a > missing daughter is not assigned by SSH properly. It could be open on the > dead server. We need to force re-assign it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7083) SSH#fixupDaughter should force re-assign missing daughter
Jimmy Xiang created HBASE-7083: -- Summary: SSH#fixupDaughter should force re-assign missing daughter Key: HBASE-7083 URL: https://issues.apache.org/jira/browse/HBASE-7083 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor In looking into flaky test TestSplitTransactionOnCluster#testShutdownSimpleFixup, I found out that a missing daughter is not assigned by SSH properly. It could be open on the dead server. We need to force re-assign it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488363#comment-13488363 ] Lars Hofhansl commented on HBASE-7066: -- So what happens when shutdown throws an IOException? Seems shutdown should not fail, no? > Some HMaster coprocessor exceptions are being swallowed in try catch blocks > --- > > Key: HBASE-7066 > URL: https://issues.apache.org/jira/browse/HBASE-7066 > Project: HBase > Issue Type: Bug > Components: Coprocessors, security >Affects Versions: 0.94.2, 0.96.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch, > HBASE-7066_trunk.patch > > > This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even > when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-7031: --- Attachment: D6129.4.patch pritamdamania updated the revision "[jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat". Reviewers: Kannan, Karthik, JIRA, mbautin Addressing comments. REVISION DETAIL https://reviews.facebook.net/D6129 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/mapred/TableInputFormat.java src/main/java/org/apache/hadoop/hbase/mapred/TableInputFormatBase.java src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReader.java src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java To: Kannan, Karthik, JIRA, mbautin, pritamdamania Cc: tjackson > Support startRow/endRow in TableInputFormat > --- > > Key: HBASE-7031 > URL: https://issues.apache.org/jira/browse/HBASE-7031 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Priority: Minor > Attachments: D6129.3.patch, D6129.4.patch > > > We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as > opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop > Streaming integration. We need to add startRow/endRow support to > TableInputFormat due to a product requirement. However, the two > TableInputFormat implementations have diverged over time. We need to make > mapred.TableInputFormat reuse some of the newer code in > mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7072) HBase-5256 breaks 0.92-0.94 compatibility
[ https://issues.apache.org/jira/browse/HBASE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-7072. --- Resolution: Fixed Hadoop Flags: Reviewed > HBase-5256 breaks 0.92-0.94 compatibility > - > > Key: HBASE-7072 > URL: https://issues.apache.org/jira/browse/HBASE-7072 > Project: HBase > Issue Type: Bug > Components: master, shell >Affects Versions: 0.92.1, 0.92.2, 0.94.0 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.92.3 > > Attachments: HBase-7072-92.patch, HBase-7072-92-v2.patch > > > HBase-5286 changes RegionLoad writable in 94.0, making it incompatible with > 92. A fix was made in HBase-5795 where a 94 client can read response from a > 92 server, but not vice versa. Currently, if a 92 client tries to do read > RegionLoad (HBase shell "status" command, or, 92 master and 94 regionserver), > it just hangs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7072) HBase-5256 breaks 0.92-0.94 compatibility
[ https://issues.apache.org/jira/browse/HBASE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7072: -- Summary: HBase-5256 breaks 0.92-0.94 compatibility (was: HBase-5256 breaks 92-94 compatibility) > HBase-5256 breaks 0.92-0.94 compatibility > - > > Key: HBASE-7072 > URL: https://issues.apache.org/jira/browse/HBASE-7072 > Project: HBase > Issue Type: Bug > Components: master, shell >Affects Versions: 0.92.1, 0.92.2, 0.94.0 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.92.3 > > Attachments: HBase-7072-92.patch, HBase-7072-92-v2.patch > > > HBase-5286 changes RegionLoad writable in 94.0, making it incompatible with > 92. A fix was made in HBase-5795 where a 94 client can read response from a > 92 server, but not vice versa. Currently, if a 92 client tries to do read > RegionLoad (HBase shell "status" command, or, 92 master and 94 regionserver), > it just hangs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7072) HBase-5256 breaks 0.92-0.94 compatibility
[ https://issues.apache.org/jira/browse/HBASE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488352#comment-13488352 ] Ted Yu commented on HBASE-7072: --- Integrated to 0.92 Thanks for the patch, Himanshu. Thanks for the review, Enis. > HBase-5256 breaks 0.92-0.94 compatibility > - > > Key: HBASE-7072 > URL: https://issues.apache.org/jira/browse/HBASE-7072 > Project: HBase > Issue Type: Bug > Components: master, shell >Affects Versions: 0.92.1, 0.92.2, 0.94.0 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.92.3 > > Attachments: HBase-7072-92.patch, HBase-7072-92-v2.patch > > > HBase-5286 changes RegionLoad writable in 94.0, making it incompatible with > 92. A fix was made in HBase-5795 where a 94 client can read response from a > 92 server, but not vice versa. Currently, if a 92 client tries to do read > RegionLoad (HBase shell "status" command, or, 92 master and 94 regionserver), > it just hangs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7076) Add test that increment/append properly integrate with MVCC
[ https://issues.apache.org/jira/browse/HBASE-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan resolved HBASE-7076. --- Resolution: Duplicate HBASE-4583 covers this. > Add test that increment/append properly integrate with MVCC > --- > > Key: HBASE-7076 > URL: https://issues.apache.org/jira/browse/HBASE-7076 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Transactions/MVCC >Affects Versions: 0.94.0, 0.96.0 >Reporter: Gregory Chanan > Fix For: 0.94.3, 0.96.0 > > > We should test that increment/append properly integrate with MVCC. > HBASE-4583 fixes the issue for increment/append. > Here is a run through of the tests: > increment: > current cell value is 10. Issue concurrently a put of 50 and an increment by > 1. Only acceptable values are 50 and 51 (not 11). > append: > current cell value is 'x'. Issue concurrently a put of 'y' and an append of > 'x'. Only acceptable values after are 'y' and 'yx' (not 'xx'). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7042) Master Coprocessor Endpoint
[ https://issues.apache.org/jira/browse/HBASE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488347#comment-13488347 ] Gary Helmling commented on HBASE-7042: -- Unfortunately this code would need to be completely reworked for the PB service support in 0.96. The Writable based CoprocessorProtocol support you're using will be completely removed for 0.96 (see HBASE-6895). Since 0.96 seems like the right target for a new feature like this, would you mind redoing this based on the CoprocessorService handling in trunk? > Master Coprocessor Endpoint > --- > > Key: HBASE-7042 > URL: https://issues.apache.org/jira/browse/HBASE-7042 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.96.0 > > Attachments: HBASE-7042_94.patch > > > Having support for a master coprocessor endpoint would enable developers to > easily extended HMaster functionality/features. As is the case for region > server grouping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7072) HBase-5256 breaks 92-94 compatibility
[ https://issues.apache.org/jira/browse/HBASE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-7072: --- Attachment: HBase-7072-92-v2.patch Comments incorporated. Thanks for review. > HBase-5256 breaks 92-94 compatibility > - > > Key: HBASE-7072 > URL: https://issues.apache.org/jira/browse/HBASE-7072 > Project: HBase > Issue Type: Bug > Components: master, shell >Affects Versions: 0.92.1, 0.92.2, 0.94.0 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.92.3 > > Attachments: HBase-7072-92.patch, HBase-7072-92-v2.patch > > > HBase-5286 changes RegionLoad writable in 94.0, making it incompatible with > 92. A fix was made in HBase-5795 where a 94 client can read response from a > 92 server, but not vice versa. Currently, if a 92 client tries to do read > RegionLoad (HBase shell "status" command, or, 92 master and 94 regionserver), > it just hangs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6894) Adding metadata to a table in the shell is both arcane and painful
[
https://issues.apache.org/jira/browse/HBASE-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488327#comment-13488327
]
Hadoop QA commented on HBASE-6894:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12551628/HBASE-6894-v4-squashed.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
85 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3206//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3206//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3206//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3206//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3206//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3206//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3206//console
This message is automatically generated.
> Adding metadata to a table in the shell is both arcane and painful
> --
>
> Key: HBASE-6894
> URL: https://issues.apache.org/jira/browse/HBASE-6894
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 0.96.0
>Reporter: stack
>Assignee: Sergey Shelukhin
> Labels: noob
> Attachments: HBASE-6894.patch, HBASE-6894.patch, HBASE-6894.patch,
> HBASE-6894-v2.patch, HBASE-6894-v3.1-squashed.patch,
> HBASE-6894-v3-squashed.patch, HBASE-6894-v4-squashed.patch
>
>
> In production we have hundreds of tables w/ whack names like 'aliaserv',
> 'ashish_bulk', 'age_gender_topics', etc. It be grand if you could look in
> master UI and see stuff like owner, eng group responsible, miscellaneous
> description, etc.
> Now, HTD has support for this; each carries a dictionary. Whats a PITA
> though is adding attributes to the dictionary. Here is what seems to work on
> trunk (though I do not trust it is doing the right thing):
> {code}
> hbase> create 'SOME_TABLENAME', {NAME => 'd', VERSION => 1, COMPRESSION =>
> 'LZO'}
> hbase> # Here is how I added metadata
> hbase> disable 'SOME_TABLENAME'
> hbase> alter 'SOME_TABLENAME', METHOD => 'table_att', OWNER => 'SOMEON',
> CONFIG => {'ENVIRONMENT' => 'BLAH BLAH', 'SIZING' => 'The size should be
> between 0-10K most of the time with new URLs coming in and getting removed as
> they are processed unless the pipeline has fallen behind', 'MISCELLANEOUS' =>
> 'Holds the list of URLs waiting to be processed in the parked page detection
> analyzer in ingestion pipeline.'}
> ...
> describe...
> enable...
> {code}
> The above doesn't work in 0.94. Complains about the CONFIG, the keyword we
> are using for the HTD dictionary.
> It works in 0.96 though I'd have to poke around some more to ensure it is
> doing the right thing.
> But this METHOD => 'table_att' stuff is really ugly can we fix it?
> And I can't add table attributes on table create seemingly.
> A little bit of thought and a bit of ruby could clean this all up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[ https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488325#comment-13488325 ] Aleksandr Shulman commented on HBASE-6904: -- That makes sense. Thanks for the explanation. > In the HBase shell, an error is thrown that states replication-related znodes > already exist > --- > > Key: HBASE-6904 > URL: https://issues.apache.org/jira/browse/HBASE-6904 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Affects Versions: 0.92.0, 0.92.1, 0.96.0 >Reporter: Aleksandr Shulman >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6904-94.patch, HBASE-6904.patch > > > On a replication-enabled cluster, querying the list_peers produces the error > lines shown below. It doesn't appear that anything is broken in terms of > functionality. > Stack trace: > hbase(main):001:0> list_peers > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/peers already exists and this is not a retry > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/rs already exists and this is not a retry > PEER ID CLUSTER KEY > 0 row(s) in 0.4650 seconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2645) HLog writer can do 1-2 sync operations after lease has been recovered for split process.
[ https://issues.apache.org/jira/browse/HBASE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488324#comment-13488324 ] Ted Yu commented on HBASE-2645: --- Another test failure: https://builds.apache.org/job/PreCommit-HBASE-Build/3204/testReport/junit/org.apache.hadoop.hbase.regionserver.wal/TestHLogSplit/testLogCannotBeWrittenOnceParsed/ > HLog writer can do 1-2 sync operations after lease has been recovered for > split process. > > > Key: HBASE-2645 > URL: https://issues.apache.org/jira/browse/HBASE-2645 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 0.90.4 >Reporter: Cosmin Lehene >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 2645.txt, 2645v2.txt, 2645v3.txt, > org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit-output.txt > > > TestHLogSplit.testLogCannotBeWrittenOnceParsed is failing. > This test starts a thread that writes one edit to the log, syncs and counts. > During this, a HLog.splitLog operation is started. splitLog recovers the log > lease before reading the log, so that the original regionserver could not > wake up and write after the split process started. > The test compares the number of edits reported by the split process and by > the writer thread. Writer thread (called zombie in the test) should report <= > than the splitLog (sync() might raise after the last edit gets written and > the edit won't get counted by zombie thread). However it appears that the > zombie counts 1-2 more edits. So it looks like it can sync without a lease. > This might be a hdfs-0.20 related issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7080) TestHFileOutputFormat.testColumnFamilyCompression failing due to UnsatisfiedLinkError
[
https://issues.apache.org/jira/browse/HBASE-7080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488320#comment-13488320
]
Hudson commented on HBASE-7080:
---
Integrated in HBase-TRUNK #3502 (See
[https://builds.apache.org/job/HBase-TRUNK/3502/])
HBASE-7080 TestHFileOutputFormat.testColumnFamilyCompression failing due to
UnsatisfiedLinkError (Revision 1404329)
Result = FAILURE
jxiang :
Files :
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
> TestHFileOutputFormat.testColumnFamilyCompression failing due to
> UnsatisfiedLinkError
> -
>
> Key: HBASE-7080
> URL: https://issues.apache.org/jira/browse/HBASE-7080
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: trunk_7080.patch
>
>
> Due to HADOOP-8756, this test fails
> {noformat}
> java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
> Method)
> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
> at
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:127)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:104)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:118)
> at
> org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:236)
> at
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.getSupportedCompressionAlgorithms(TestHFileOutputFormat.java:649)
> at
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testColumnFamilyCompression(TestHFileOutputFormat.java:571)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[
https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488319#comment-13488319
]
Gregory Chanan commented on HBASE-6904:
---
There is such a call but it doesn't really make things any better.
Let's say you used that call...you'd do something like this:
{code}
if(!zk.exists(znode)) {
zk.create(znode);
}
{code}
But what if another client does the same thing at the same time? (remember, we
just want the znode to exist, don't care who creates it). Well, one of the
create calls will get a NodeExistsException, so you need the try/catch anyway
and calling exists hasn't saved you anything.
> In the HBase shell, an error is thrown that states replication-related znodes
> already exist
> ---
>
> Key: HBASE-6904
> URL: https://issues.apache.org/jira/browse/HBASE-6904
> Project: HBase
> Issue Type: Bug
> Components: Replication, Zookeeper
>Affects Versions: 0.92.0, 0.92.1, 0.96.0
>Reporter: Aleksandr Shulman
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.94.3, 0.96.0
>
> Attachments: HBASE-6904-94.patch, HBASE-6904.patch
>
>
> On a replication-enabled cluster, querying the list_peers produces the error
> lines shown below. It doesn't appear that anything is broken in terms of
> functionality.
> Stack trace:
> hbase(main):001:0> list_peers
> 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node
> /hbase/replication/peers already exists and this is not a retry
> 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node
> /hbase/replication/rs already exists and this is not a retry
> PEER ID CLUSTER KEY
> 0 row(s) in 0.4650 seconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[ https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488309#comment-13488309 ] Aleksandr Shulman commented on HBASE-6904: -- Ah ok, thanks for clarifying. I don't know if there is such a call, but perhaps we should check if the node exists ourselves instead of relying on an NodeExistsException. This makes it seem fragile. If there is no such call (e.g. ZK.existsNode(...) ), then I suppose this is the way to do it. > In the HBase shell, an error is thrown that states replication-related znodes > already exist > --- > > Key: HBASE-6904 > URL: https://issues.apache.org/jira/browse/HBASE-6904 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Affects Versions: 0.92.0, 0.92.1, 0.96.0 >Reporter: Aleksandr Shulman >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6904-94.patch, HBASE-6904.patch > > > On a replication-enabled cluster, querying the list_peers produces the error > lines shown below. It doesn't appear that anything is broken in terms of > functionality. > Stack trace: > hbase(main):001:0> list_peers > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/peers already exists and this is not a retry > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/rs already exists and this is not a retry > PEER ID CLUSTER KEY > 0 row(s) in 0.4650 seconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[ https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-6904: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to 0.94 and trunk. > In the HBase shell, an error is thrown that states replication-related znodes > already exist > --- > > Key: HBASE-6904 > URL: https://issues.apache.org/jira/browse/HBASE-6904 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Affects Versions: 0.92.0, 0.92.1, 0.96.0 >Reporter: Aleksandr Shulman >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6904-94.patch, HBASE-6904.patch > > > On a replication-enabled cluster, querying the list_peers produces the error > lines shown below. It doesn't appear that anything is broken in terms of > functionality. > Stack trace: > hbase(main):001:0> list_peers > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/peers already exists and this is not a retry > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/rs already exists and this is not a retry > PEER ID CLUSTER KEY > 0 row(s) in 0.4650 seconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[
https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488300#comment-13488300
]
Gregory Chanan commented on HBASE-6904:
---
It's not a meaningless exception and it's not an error.
ZooKeeper.create will throw a NodeExists exception if the znode you are trying
to create already exists. This code just wants the znode to exist so it
essentially does this...
{code}
try {
ZK.create()
} catch (NodeExistsException) {
// okay
}
{code}
> In the HBase shell, an error is thrown that states replication-related znodes
> already exist
> ---
>
> Key: HBASE-6904
> URL: https://issues.apache.org/jira/browse/HBASE-6904
> Project: HBase
> Issue Type: Bug
> Components: Replication, Zookeeper
>Affects Versions: 0.92.0, 0.92.1, 0.96.0
>Reporter: Aleksandr Shulman
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.94.3, 0.96.0
>
> Attachments: HBASE-6904-94.patch, HBASE-6904.patch
>
>
> On a replication-enabled cluster, querying the list_peers produces the error
> lines shown below. It doesn't appear that anything is broken in terms of
> functionality.
> Stack trace:
> hbase(main):001:0> list_peers
> 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node
> /hbase/replication/peers already exists and this is not a retry
> 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node
> /hbase/replication/rs already exists and this is not a retry
> PEER ID CLUSTER KEY
> 0 row(s) in 0.4650 seconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7077) Test for: CheckAndPut should properly read MVCC
[ https://issues.apache.org/jira/browse/HBASE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-7077. -- Resolution: Fixed Committed to 0.94 and 0.96. Thanks for the nice test Ram. (I change the test name to TestHBase7051) > Test for: CheckAndPut should properly read MVCC > --- > > Key: HBASE-7077 > URL: https://issues.apache.org/jira/browse/HBASE-7077 > Project: HBase > Issue Type: Sub-task >Reporter: Gregory Chanan > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-7071.patch, HBASE-7071_testcasewithassert.patch > > > checkAndPut should integrate with MVCC, similar to how HBASE-4583 fixed > appends and increments. > Also need a test, here's one we could use (originally proposed in HBASE-7051): > The current value of some cell is 10. > I issue two concurrent requests: > A) a check and put where check value = 10, put value = 11 > B) a put where put value = 50 > The only result at the end of these operations that seems reasonable to me is > the value of the cell being 50. If A occurred first (ACID wise), then our > values go 10->11->50. If B occurred first, then our values go 10->50 (and the > checkAndPut fails) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7082) TestHFileCleaner#testHFileCleaning fails due to cleaner is reset
[
https://issues.apache.org/jira/browse/HBASE-7082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488298#comment-13488298
]
Hadoop QA commented on HBASE-7082:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12551624/trunk-7082.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
85 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3204//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3204//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3204//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3204//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3204//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3204//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3204//console
This message is automatically generated.
> TestHFileCleaner#testHFileCleaning fails due to cleaner is reset
>
>
> Key: HBASE-7082
> URL: https://issues.apache.org/jira/browse/HBASE-7082
> Project: HBase
> Issue Type: Test
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Trivial
> Attachments: trunk-7082.patch
>
>
> TestHFileCleaner#testHFileCleaning fails if it runs after
> testRemovesEmptyDirectories which resets the cleaner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7077) Test for: CheckAndPut should properly read MVCC
[ https://issues.apache.org/jira/browse/HBASE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7077: - Fix Version/s: 0.94.3 > Test for: CheckAndPut should properly read MVCC > --- > > Key: HBASE-7077 > URL: https://issues.apache.org/jira/browse/HBASE-7077 > Project: HBase > Issue Type: Sub-task >Reporter: Gregory Chanan > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-7071.patch, HBASE-7071_testcasewithassert.patch > > > checkAndPut should integrate with MVCC, similar to how HBASE-4583 fixed > appends and increments. > Also need a test, here's one we could use (originally proposed in HBASE-7051): > The current value of some cell is 10. > I issue two concurrent requests: > A) a check and put where check value = 10, put value = 11 > B) a put where put value = 50 > The only result at the end of these operations that seems reasonable to me is > the value of the cell being 50. If A occurred first (ACID wise), then our > values go 10->11->50. If B occurred first, then our values go 10->50 (and the > checkAndPut fails) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7072) HBase-5256 breaks 92-94 compatibility
[
https://issues.apache.org/jira/browse/HBASE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488294#comment-13488294
]
Ted Yu commented on HBASE-7072:
---
Please fix the spelling for method name:
{code}
+public void readFiels94(DataInput in) throws IOException {
{code}
Otherwise patch looks good.
> HBase-5256 breaks 92-94 compatibility
> -
>
> Key: HBASE-7072
> URL: https://issues.apache.org/jira/browse/HBASE-7072
> Project: HBase
> Issue Type: Bug
> Components: master, shell
>Affects Versions: 0.92.1, 0.92.2, 0.94.0
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.92.3
>
> Attachments: HBase-7072-92.patch
>
>
> HBase-5286 changes RegionLoad writable in 94.0, making it incompatible with
> 92. A fix was made in HBase-5795 where a 94 client can read response from a
> 92 server, but not vice versa. Currently, if a 92 client tries to do read
> RegionLoad (HBase shell "status" command, or, 92 master and 94 regionserver),
> it just hangs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7077) Test for: CheckAndPut should properly read MVCC
[ https://issues.apache.org/jira/browse/HBASE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7077: - Summary: Test for: CheckAndPut should properly read MVCC (was: Test for: checkAndPut should properly integrate with MVCC) > Test for: CheckAndPut should properly read MVCC > --- > > Key: HBASE-7077 > URL: https://issues.apache.org/jira/browse/HBASE-7077 > Project: HBase > Issue Type: Sub-task >Reporter: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7071.patch, HBASE-7071_testcasewithassert.patch > > > checkAndPut should integrate with MVCC, similar to how HBASE-4583 fixed > appends and increments. > Also need a test, here's one we could use (originally proposed in HBASE-7051): > The current value of some cell is 10. > I issue two concurrent requests: > A) a check and put where check value = 10, put value = 11 > B) a put where put value = 50 > The only result at the end of these operations that seems reasonable to me is > the value of the cell being 50. If A occurred first (ACID wise), then our > values go 10->11->50. If B occurred first, then our values go 10->50 (and the > checkAndPut fails) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[ https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488291#comment-13488291 ] Aleksandr Shulman commented on HBASE-6904: -- Well, I guess I'm as much concerned with whether the message is displayed at the shell level (it's not great, we should fix it regardless). I'm more concerned as to why we hit this failure mode to begin with. We should go down to where the code throws the exception and investigate. If it's a spurious exception, we need to get rid of it. If it's a meaningful exception, we need to see why it occurs every time the shell starts (and stop it from happening). > In the HBase shell, an error is thrown that states replication-related znodes > already exist > --- > > Key: HBASE-6904 > URL: https://issues.apache.org/jira/browse/HBASE-6904 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Affects Versions: 0.92.0, 0.92.1, 0.96.0 >Reporter: Aleksandr Shulman >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6904-94.patch, HBASE-6904.patch > > > On a replication-enabled cluster, querying the list_peers produces the error > lines shown below. It doesn't appear that anything is broken in terms of > functionality. > Stack trace: > hbase(main):001:0> list_peers > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/peers already exists and this is not a retry > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/rs already exists and this is not a retry > PEER ID CLUSTER KEY > 0 row(s) in 0.4650 seconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7071) Allow for user defined Data in HBase Table & ColumnFamily definitions
[ https://issues.apache.org/jira/browse/HBASE-7071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488288#comment-13488288 ] stack commented on HBASE-7071: -- [~huhlig] Have you tried it using the CONFIG option Sergey notes above? Would be interested in your experience. > Allow for user defined Data in HBase Table & ColumnFamily definitions > - > > Key: HBASE-7071 > URL: https://issues.apache.org/jira/browse/HBASE-7071 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 0.94.2 >Reporter: Hans Uhlig > > Allow for arbitrary user data in HBase Table & Column Family definitions. > Use case: > Allowing multiple applications to dynamically access data in HBase tables and > fetch schema information from table or column family directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7051) CheckAndPut should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-7051:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to 0.94 and 0.96.
Thanks for bringing up the issue and Gregory and the review.
> CheckAndPut should properly read MVCC
> -
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7051.txt
>
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7072) HBase-5256 breaks 92-94 compatibility
[
https://issues.apache.org/jira/browse/HBASE-7072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488285#comment-13488285
]
Enis Soztutar commented on HBASE-7072:
--
It is just unfortunate that we had HBASE-5256 committed to 0.94. At this point
we cannot revert it since it is already released. It seems that the only thing
we can do for 92 is to apply this patch.
+1 on the patch. My only nit is to remove the line.
{code}
+ // super.readFields(in);
{code}
> HBase-5256 breaks 92-94 compatibility
> -
>
> Key: HBASE-7072
> URL: https://issues.apache.org/jira/browse/HBASE-7072
> Project: HBase
> Issue Type: Bug
> Components: master, shell
>Affects Versions: 0.92.1, 0.92.2, 0.94.0
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.92.3
>
> Attachments: HBase-7072-92.patch
>
>
> HBase-5286 changes RegionLoad writable in 94.0, making it incompatible with
> 92. A fix was made in HBase-5795 where a 94 client can read response from a
> 92 server, but not vice versa. Currently, if a 92 client tries to do read
> RegionLoad (HBase shell "status" command, or, 92 master and 94 regionserver),
> it just hangs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-1328) Evangelize HBase in Media
[ https://issues.apache.org/jira/browse/HBASE-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-1328. -- Resolution: Won't Fix Resolving as won't fix inside. Lets open specific issues rather than this general umbrella issue. > Evangelize HBase in Media > - > > Key: HBASE-1328 > URL: https://issues.apache.org/jira/browse/HBASE-1328 > Project: HBase > Issue Type: Task >Reporter: Lars George >Priority: Critical > > We need to spread the word more on HBase to advance adoption. > I took the first step and create a HBase Twitter account that talks about > itself as it "sees" itself being mentioned on the intertubes. Here the URL: > http://twitter.com/HBase > We should discuss further means of a combines effort given our limited > advertisement budget. > While I created the account for Twitter, I suggest we share the details and > set the email address to a central account, so that more than one person can > post updates. BTW, other projects on Apache.org follow the same approach, so > this is more or less a working concept. And avoids someone else hogging the > HBase account name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7051) CheckAndPut should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-7051:
-
Summary: CheckAndPut should properly read MVCC (was: Read/Updates
(increment,checkAndPut) should properly read MVCC)
> CheckAndPut should properly read MVCC
> -
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7051.txt
>
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6305) TestLocalHBaseCluster hangs with hadoop 2.0/0.23 builds.
[
https://issues.apache.org/jira/browse/HBASE-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488280#comment-13488280
]
Jimmy Xiang commented on HBASE-6305:
+1
> TestLocalHBaseCluster hangs with hadoop 2.0/0.23 builds.
>
>
> Key: HBASE-6305
> URL: https://issues.apache.org/jira/browse/HBASE-6305
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Affects Versions: 0.92.2, 0.94.1
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.94.3
>
> Attachments: hbase-6305-94.patch, HBASE-6305-94-v2.patch,
> HBASE-6305-94-v2.patch, HBASE-6305-v1.patch
>
>
> trunk: mvn clean test -Dhadoop.profile=2.0 -Dtest=TestLocalHBaseCluster
> 0.94: mvn clean test -Dhadoop.profile=23 -Dtest=TestLocalHBaseCluster
> {code}
> testLocalHBaseCluster(org.apache.hadoop.hbase.TestLocalHBaseCluster) Time
> elapsed: 0.022 sec <<< ERROR!
> java.lang.RuntimeException: Master not initialized after 200 seconds
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:208)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:424)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:66)
> ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[
https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488281#comment-13488281
]
Hadoop QA commented on HBASE-6904:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12551633/HBASE-6904-94.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3207//console
This message is automatically generated.
> In the HBase shell, an error is thrown that states replication-related znodes
> already exist
> ---
>
> Key: HBASE-6904
> URL: https://issues.apache.org/jira/browse/HBASE-6904
> Project: HBase
> Issue Type: Bug
> Components: Replication, Zookeeper
>Affects Versions: 0.92.0, 0.92.1, 0.96.0
>Reporter: Aleksandr Shulman
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.94.3, 0.96.0
>
> Attachments: HBASE-6904-94.patch, HBASE-6904.patch
>
>
> On a replication-enabled cluster, querying the list_peers produces the error
> lines shown below. It doesn't appear that anything is broken in terms of
> functionality.
> Stack trace:
> hbase(main):001:0> list_peers
> 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node
> /hbase/replication/peers already exists and this is not a retry
> 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node
> /hbase/replication/rs already exists and this is not a retry
> PEER ID CLUSTER KEY
> 0 row(s) in 0.4650 seconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7077) Test for: checkAndPut should properly integrate with MVCC
[ https://issues.apache.org/jira/browse/HBASE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7077: - Summary: Test for: checkAndPut should properly integrate with MVCC (was: checkAndPut should properly integrate with MVCC) > Test for: checkAndPut should properly integrate with MVCC > - > > Key: HBASE-7077 > URL: https://issues.apache.org/jira/browse/HBASE-7077 > Project: HBase > Issue Type: Sub-task >Reporter: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7071.patch, HBASE-7071_testcasewithassert.patch > > > checkAndPut should integrate with MVCC, similar to how HBASE-4583 fixed > appends and increments. > Also need a test, here's one we could use (originally proposed in HBASE-7051): > The current value of some cell is 10. > I issue two concurrent requests: > A) a check and put where check value = 10, put value = 11 > B) a put where put value = 50 > The only result at the end of these operations that seems reasonable to me is > the value of the cell being 50. If A occurred first (ACID wise), then our > values go 10->11->50. If B occurred first, then our values go 10->50 (and the > checkAndPut fails) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7031) Support startRow/endRow in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488277#comment-13488277 ] Phabricator commented on HBASE-7031: tjackson has commented on the revision "[jira] [HBASE-7031] [89-fb] Add startRow/endRow to TableInputFormat". A few little things, but otherwise this looks good. Could you grab someone more familiar with HBase/MR to accept? INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/mapred/TableInputFormatBase.java:132 ultranit: space after comma. src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java:73 I'd make the field the concrete type, since it's private. This will let Java see what methods are final and avoid the virtual invocations costs. src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java:227 Can't you just declare the method ##synchronized##? REVISION DETAIL https://reviews.facebook.net/D6129 To: Kannan, Karthik, JIRA, mbautin, pritamdamania Cc: tjackson > Support startRow/endRow in TableInputFormat > --- > > Key: HBASE-7031 > URL: https://issues.apache.org/jira/browse/HBASE-7031 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Priority: Minor > Attachments: D6129.3.patch > > > We are still using org.apache.hadoop.hbase.mapred.TableInputFormat (as > opposed to org.apache.hadoop.hbase.mapreduce.TableInputFormat) for Hadoop > Streaming integration. We need to add startRow/endRow support to > TableInputFormat due to a product requirement. However, the two > TableInputFormat implementations have diverged over time. We need to make > mapred.TableInputFormat reuse some of the newer code in > mapreduce.TableInputFormat, and add startRow/endRow support to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[
https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488273#comment-13488273
]
Gregory Chanan commented on HBASE-6904:
---
No, it's not a problem. I think my above comments address your concerns:
{quote}This error log is unnecessary, because the code already throws an
exception. The code that calls it can just decide if it wants to reprint the
error message or not by catching/rethrowing the exception. I looked through the
calls, which all originate in ZKUtil, and I think it's fine to just bump this
down to "info" level. All the calls in ZKUtil either rethrow the exception or
have semantics where getting a NodeExists exception is not a problem.{quote}
> In the HBase shell, an error is thrown that states replication-related znodes
> already exist
> ---
>
> Key: HBASE-6904
> URL: https://issues.apache.org/jira/browse/HBASE-6904
> Project: HBase
> Issue Type: Bug
> Components: Replication, Zookeeper
>Affects Versions: 0.92.0, 0.92.1, 0.96.0
>Reporter: Aleksandr Shulman
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.94.3, 0.96.0
>
> Attachments: HBASE-6904-94.patch, HBASE-6904.patch
>
>
> On a replication-enabled cluster, querying the list_peers produces the error
> lines shown below. It doesn't appear that anything is broken in terms of
> functionality.
> Stack trace:
> hbase(main):001:0> list_peers
> 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node
> /hbase/replication/peers already exists and this is not a retry
> 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node
> /hbase/replication/rs already exists and this is not a retry
> PEER ID CLUSTER KEY
> 0 row(s) in 0.4650 seconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[ https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488274#comment-13488274 ] Gregory Chanan commented on HBASE-6904: --- Going to commit to trunk/0.94. > In the HBase shell, an error is thrown that states replication-related znodes > already exist > --- > > Key: HBASE-6904 > URL: https://issues.apache.org/jira/browse/HBASE-6904 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Affects Versions: 0.92.0, 0.92.1, 0.96.0 >Reporter: Aleksandr Shulman >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6904-94.patch, HBASE-6904.patch > > > On a replication-enabled cluster, querying the list_peers produces the error > lines shown below. It doesn't appear that anything is broken in terms of > functionality. > Stack trace: > hbase(main):001:0> list_peers > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/peers already exists and this is not a retry > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/rs already exists and this is not a retry > PEER ID CLUSTER KEY > 0 row(s) in 0.4650 seconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[ https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-6904: -- Attachment: HBASE-6904-94.patch Patch for 94. > In the HBase shell, an error is thrown that states replication-related znodes > already exist > --- > > Key: HBASE-6904 > URL: https://issues.apache.org/jira/browse/HBASE-6904 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Affects Versions: 0.92.0, 0.92.1, 0.96.0 >Reporter: Aleksandr Shulman >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6904-94.patch, HBASE-6904.patch > > > On a replication-enabled cluster, querying the list_peers produces the error > lines shown below. It doesn't appear that anything is broken in terms of > functionality. > Stack trace: > hbase(main):001:0> list_peers > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/peers already exists and this is not a retry > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/rs already exists and this is not a retry > PEER ID CLUSTER KEY > 0 row(s) in 0.4650 seconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7077) checkAndPut should properly integrate with MVCC
[ https://issues.apache.org/jira/browse/HBASE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488270#comment-13488270 ] Lars Hofhansl commented on HBASE-7077: -- I'll make that change and commit the test. I verified that it validates the fix in HBASE-7051 correctly. > checkAndPut should properly integrate with MVCC > --- > > Key: HBASE-7077 > URL: https://issues.apache.org/jira/browse/HBASE-7077 > Project: HBase > Issue Type: Sub-task >Reporter: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7071.patch, HBASE-7071_testcasewithassert.patch > > > checkAndPut should integrate with MVCC, similar to how HBASE-4583 fixed > appends and increments. > Also need a test, here's one we could use (originally proposed in HBASE-7051): > The current value of some cell is 10. > I issue two concurrent requests: > A) a check and put where check value = 10, put value = 11 > B) a put where put value = 50 > The only result at the end of these operations that seems reasonable to me is > the value of the cell being 50. If A occurred first (ACID wise), then our > values go 10->11->50. If B occurred first, then our values go 10->50 (and the > checkAndPut fails) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6305) TestLocalHBaseCluster hangs with hadoop 2.0/0.23 builds.
[
https://issues.apache.org/jira/browse/HBASE-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488267#comment-13488267
]
Hadoop QA commented on HBASE-6305:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12551631/HBASE-6305-94-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3205//console
This message is automatically generated.
> TestLocalHBaseCluster hangs with hadoop 2.0/0.23 builds.
>
>
> Key: HBASE-6305
> URL: https://issues.apache.org/jira/browse/HBASE-6305
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Affects Versions: 0.92.2, 0.94.1
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.94.3
>
> Attachments: hbase-6305-94.patch, HBASE-6305-94-v2.patch,
> HBASE-6305-94-v2.patch, HBASE-6305-v1.patch
>
>
> trunk: mvn clean test -Dhadoop.profile=2.0 -Dtest=TestLocalHBaseCluster
> 0.94: mvn clean test -Dhadoop.profile=23 -Dtest=TestLocalHBaseCluster
> {code}
> testLocalHBaseCluster(org.apache.hadoop.hbase.TestLocalHBaseCluster) Time
> elapsed: 0.022 sec <<< ERROR!
> java.lang.RuntimeException: Master not initialized after 200 seconds
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:208)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:424)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:66)
> ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7051) Read/Updates (increment,checkAndPut) should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488268#comment-13488268
]
Lars Hofhansl commented on HBASE-7051:
--
Sigh... I also meant HBASE-7077. So this one + HBASE-7077. :)
> Read/Updates (increment,checkAndPut) should properly read MVCC
> --
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7051.txt
>
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488266#comment-13488266 ] Sergey Shelukhin commented on HBASE-7055: - I think it's too big/active to be called a policy... http://www.boost.org/community/generic_programming.html#policy > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6305) TestLocalHBaseCluster hangs with hadoop 2.0/0.23 builds.
[
https://issues.apache.org/jira/browse/HBASE-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488264#comment-13488264
]
Himanshu Vashishtha commented on HBASE-6305:
It passes on hadoop 1 and hadoop 2, both.
> TestLocalHBaseCluster hangs with hadoop 2.0/0.23 builds.
>
>
> Key: HBASE-6305
> URL: https://issues.apache.org/jira/browse/HBASE-6305
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Affects Versions: 0.92.2, 0.94.1
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.94.3
>
> Attachments: hbase-6305-94.patch, HBASE-6305-94-v2.patch,
> HBASE-6305-94-v2.patch, HBASE-6305-v1.patch
>
>
> trunk: mvn clean test -Dhadoop.profile=2.0 -Dtest=TestLocalHBaseCluster
> 0.94: mvn clean test -Dhadoop.profile=23 -Dtest=TestLocalHBaseCluster
> {code}
> testLocalHBaseCluster(org.apache.hadoop.hbase.TestLocalHBaseCluster) Time
> elapsed: 0.022 sec <<< ERROR!
> java.lang.RuntimeException: Master not initialized after 200 seconds
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:208)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:424)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:66)
> ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6305) TestLocalHBaseCluster hangs with hadoop 2.0/0.23 builds.
[
https://issues.apache.org/jira/browse/HBASE-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Vashishtha updated HBASE-6305:
---
Attachment: HBASE-6305-94-v2.patch
Ok.
The makeQualified(FileSystem) argument is deprecated. I used the newer method
instead.
> TestLocalHBaseCluster hangs with hadoop 2.0/0.23 builds.
>
>
> Key: HBASE-6305
> URL: https://issues.apache.org/jira/browse/HBASE-6305
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Affects Versions: 0.92.2, 0.94.1
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.94.3
>
> Attachments: hbase-6305-94.patch, HBASE-6305-94-v2.patch,
> HBASE-6305-94-v2.patch, HBASE-6305-v1.patch
>
>
> trunk: mvn clean test -Dhadoop.profile=2.0 -Dtest=TestLocalHBaseCluster
> 0.94: mvn clean test -Dhadoop.profile=23 -Dtest=TestLocalHBaseCluster
> {code}
> testLocalHBaseCluster(org.apache.hadoop.hbase.TestLocalHBaseCluster) Time
> elapsed: 0.022 sec <<< ERROR!
> java.lang.RuntimeException: Master not initialized after 200 seconds
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:208)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:424)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:66)
> ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6894) Adding metadata to a table in the shell is both arcane and painful
[
https://issues.apache.org/jira/browse/HBASE-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HBASE-6894:
Attachment: HBASE-6894-v4-squashed.patch
> Adding metadata to a table in the shell is both arcane and painful
> --
>
> Key: HBASE-6894
> URL: https://issues.apache.org/jira/browse/HBASE-6894
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 0.96.0
>Reporter: stack
>Assignee: Sergey Shelukhin
> Labels: noob
> Attachments: HBASE-6894.patch, HBASE-6894.patch, HBASE-6894.patch,
> HBASE-6894-v2.patch, HBASE-6894-v3.1-squashed.patch,
> HBASE-6894-v3-squashed.patch, HBASE-6894-v4-squashed.patch
>
>
> In production we have hundreds of tables w/ whack names like 'aliaserv',
> 'ashish_bulk', 'age_gender_topics', etc. It be grand if you could look in
> master UI and see stuff like owner, eng group responsible, miscellaneous
> description, etc.
> Now, HTD has support for this; each carries a dictionary. Whats a PITA
> though is adding attributes to the dictionary. Here is what seems to work on
> trunk (though I do not trust it is doing the right thing):
> {code}
> hbase> create 'SOME_TABLENAME', {NAME => 'd', VERSION => 1, COMPRESSION =>
> 'LZO'}
> hbase> # Here is how I added metadata
> hbase> disable 'SOME_TABLENAME'
> hbase> alter 'SOME_TABLENAME', METHOD => 'table_att', OWNER => 'SOMEON',
> CONFIG => {'ENVIRONMENT' => 'BLAH BLAH', 'SIZING' => 'The size should be
> between 0-10K most of the time with new URLs coming in and getting removed as
> they are processed unless the pipeline has fallen behind', 'MISCELLANEOUS' =>
> 'Holds the list of URLs waiting to be processed in the parked page detection
> analyzer in ingestion pipeline.'}
> ...
> describe...
> enable...
> {code}
> The above doesn't work in 0.94. Complains about the CONFIG, the keyword we
> are using for the HTD dictionary.
> It works in 0.96 though I'd have to poke around some more to ensure it is
> doing the right thing.
> But this METHOD => 'table_att' stuff is really ugly can we fix it?
> And I can't add table attributes on table create seemingly.
> A little bit of thought and a bit of ruby could clean this all up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[ https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488242#comment-13488242 ] Aleksandr Shulman commented on HBASE-6904: -- Is it actually a problem that the replication znodes exist? From the comments, it would seem not. However, the author of this piece of code seemed to think it was a problem. How are we reconciling those two things? > In the HBase shell, an error is thrown that states replication-related znodes > already exist > --- > > Key: HBASE-6904 > URL: https://issues.apache.org/jira/browse/HBASE-6904 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Affects Versions: 0.92.0, 0.92.1, 0.96.0 >Reporter: Aleksandr Shulman >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6904.patch > > > On a replication-enabled cluster, querying the list_peers produces the error > lines shown below. It doesn't appear that anything is broken in terms of > functionality. > Stack trace: > hbase(main):001:0> list_peers > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/peers already exists and this is not a retry > 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node > /hbase/replication/rs already exists and this is not a retry > PEER ID CLUSTER KEY > 0 row(s) in 0.4650 seconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488238#comment-13488238 ] Ted Yu commented on HBASE-7055: --- Can we keep the class name CompactionManager so that comparison between trunk and 0.89-fb is easier ? > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7082) TestHFileCleaner#testHFileCleaning fails due to cleaner is reset
[ https://issues.apache.org/jira/browse/HBASE-7082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7082: --- Status: Patch Available (was: Open) > TestHFileCleaner#testHFileCleaning fails due to cleaner is reset > > > Key: HBASE-7082 > URL: https://issues.apache.org/jira/browse/HBASE-7082 > Project: HBase > Issue Type: Test >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > Attachments: trunk-7082.patch > > > TestHFileCleaner#testHFileCleaning fails if it runs after > testRemovesEmptyDirectories which resets the cleaner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7082) TestHFileCleaner#testHFileCleaning fails due to cleaner is reset
[ https://issues.apache.org/jira/browse/HBASE-7082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7082: --- Attachment: trunk-7082.patch > TestHFileCleaner#testHFileCleaning fails due to cleaner is reset > > > Key: HBASE-7082 > URL: https://issues.apache.org/jira/browse/HBASE-7082 > Project: HBase > Issue Type: Test >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > Attachments: trunk-7082.patch > > > TestHFileCleaner#testHFileCleaning fails if it runs after > testRemovesEmptyDirectories which resets the cleaner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7082) TestHFileCleaner#testHFileCleaning fails due to cleaner is reset
Jimmy Xiang created HBASE-7082: -- Summary: TestHFileCleaner#testHFileCleaning fails due to cleaner is reset Key: HBASE-7082 URL: https://issues.apache.org/jira/browse/HBASE-7082 Project: HBase Issue Type: Test Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Trivial Attachments: trunk-7082.patch TestHFileCleaner#testHFileCleaning fails if it runs after testRemovesEmptyDirectories which resets the cleaner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488223#comment-13488223 ] Enis Soztutar commented on HBASE-7055: -- bq. +1 to separate out the compaction policy refactoring - that bit is nice. +1 as well. Also can we rename CompactionManager -> CompactionPolicy > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488216#comment-13488216 ] Todd Lipcon commented on HBASE-7055: +1 to separate out the compaction policy refactoring - that bit is nice. > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488215#comment-13488215 ] Francis Liu commented on HBASE-7066: Ted, I did not change the signatures for HMaster in 0.94 as Ram suggested. So that patch should be ok. > Some HMaster coprocessor exceptions are being swallowed in try catch blocks > --- > > Key: HBASE-7066 > URL: https://issues.apache.org/jira/browse/HBASE-7066 > Project: HBase > Issue Type: Bug > Components: Coprocessors, security >Affects Versions: 0.94.2, 0.96.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch, > HBASE-7066_trunk.patch > > > This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even > when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7066) Some HMaster coprocessor exceptions are being swallowed in try catch blocks
[ https://issues.apache.org/jira/browse/HBASE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488198#comment-13488198 ] Francis Liu commented on HBASE-7066: Stack, IMHO it really depends on what the coprocessor wants. In the context of security, it'd be safer to err on the side of caution if the security module is unable to authorize a request, as this behavior may be exploited by a malicious user. In general though we probably need a means for the coprocessor to communicate what can and can't be ignored. > Some HMaster coprocessor exceptions are being swallowed in try catch blocks > --- > > Key: HBASE-7066 > URL: https://issues.apache.org/jira/browse/HBASE-7066 > Project: HBase > Issue Type: Bug > Components: Coprocessors, security >Affects Versions: 0.94.2, 0.96.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-7066_94.patch, HBASE-7066_trunk.patch, > HBASE-7066_trunk.patch > > > This is causing HMaster.shutdown() and HMaster.stopMaster() to succeed even > when an AccessDeniedException is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488196#comment-13488196 ] Sergey Shelukhin commented on HBASE-7055: - Now that I think of it, any objections to porting the refactoring part (e.g. CompactionManager from (H)Store), without configurability or tiers, so that it would be easier to implement whatever approach is deemed best? > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488195#comment-13488195 ] Sergey Shelukhin commented on HBASE-7055: - We discussed with Enis and it seems like this kind of configuration should be stored on table and column family level; either with some explicit system parameters, or as a blob (similar to the storing-HCat/etc.-metadata ideas), depending on how we thing about different compaction policies (system provides a set e.g. the tiered one vs the default one, or they are more pluggable/user-changeable). Another issue is that the new policy is configured on by default in the xml config (I missed that), which seems premature since use cases are not 100% clear, in a sense that people might have different scenarios for what constitutes old data/hot data/lots of data, so it's hard to come up with reasonable tier defaults. I will recall the patch for now... > port HBASE-6371 tier-based compaction from 0.89-fb to trunk > --- > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7077) checkAndPut should properly integrate with MVCC
[ https://issues.apache.org/jira/browse/HBASE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488192#comment-13488192 ] Gregory Chanan commented on HBASE-7077: --- The test references HBASE-7071, but that is not related. Change it to reference HBASE-7051? > checkAndPut should properly integrate with MVCC > --- > > Key: HBASE-7077 > URL: https://issues.apache.org/jira/browse/HBASE-7077 > Project: HBase > Issue Type: Sub-task >Reporter: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7071.patch, HBASE-7071_testcasewithassert.patch > > > checkAndPut should integrate with MVCC, similar to how HBASE-4583 fixed > appends and increments. > Also need a test, here's one we could use (originally proposed in HBASE-7051): > The current value of some cell is 10. > I issue two concurrent requests: > A) a check and put where check value = 10, put value = 11 > B) a put where put value = 50 > The only result at the end of these operations that seems reasonable to me is > the value of the cell being 50. If A occurred first (ACID wise), then our > values go 10->11->50. If B occurred first, then our values go 10->50 (and the > checkAndPut fails) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7080) TestHFileOutputFormat.testColumnFamilyCompression failing due to UnsatisfiedLinkError
[
https://issues.apache.org/jira/browse/HBASE-7080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-7080:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Integrated into trunk. Thanks Greg for reviewing it.
> TestHFileOutputFormat.testColumnFamilyCompression failing due to
> UnsatisfiedLinkError
> -
>
> Key: HBASE-7080
> URL: https://issues.apache.org/jira/browse/HBASE-7080
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: trunk_7080.patch
>
>
> Due to HADOOP-8756, this test fails
> {noformat}
> java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
> Method)
> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
> at
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:127)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:104)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:118)
> at
> org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:236)
> at
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.getSupportedCompressionAlgorithms(TestHFileOutputFormat.java:649)
> at
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testColumnFamilyCompression(TestHFileOutputFormat.java:571)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7051) Read/Updates (increment,checkAndPut) should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488176#comment-13488176
]
Gregory Chanan commented on HBASE-7051:
---
I'm +1 on this. I haven't looked at HBASE-7071 enough to +1 it. Want me to?
> Read/Updates (increment,checkAndPut) should properly read MVCC
> --
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7051.txt
>
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
