[jira] [Created] (HBASE-27117) Update the method comments for RegionServerAccounting
Tao Li created HBASE-27117: -- Summary: Update the method comments for RegionServerAccounting Key: HBASE-27117 URL: https://issues.apache.org/jira/browse/HBASE-27117 Project: HBase Issue Type: Bug Reporter: Tao Li Assignee: Tao Li After HBASE-15787, the return value type of RegionServerAccounting#isAboveHighWaterMark and RegionServerAccounting#isAboveLowWaterMark are no longer boolean. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27116) Non-interactive missing for the hbase-shell
Dong0829 created HBASE-27116: Summary: Non-interactive missing for the hbase-shell Key: HBASE-27116 URL: https://issues.apache.org/jira/browse/HBASE-27116 Project: HBase Issue Type: Bug Components: shell Affects Versions: 2.4.12 Reporter: Dong0829 During test 2.4.12, we found a lot of our integration testing failure for the non-interactive mode in hbase-shell, looks like in 2.4.12, we removed the "shell.eval_io" for the non-interactive shell, this caused the input/output handle behavior for shell is the same which caused unexpected error, for example: ``` echo -e "disable_all '.*'\ny" | hbase shell -n =Old version 2.4.4= t1 Disable the above 1 tables (y/n)?1 tables successfully disabled Took 0.7811 seconds =New version 2.4.12= hbase:001:0> disable_all '.*'t1 test_tableDisable the above 2 tables (y/n)? Took 0.5270 seconds NoMethodError: undefined method `chomp' for nil:NilClass command at uri:classloader:/shell/commands/disable_all.rb:41 command_safe at uri:classloader:/shell/commands.rb:49 translate_hbase_exceptions at uri:classloader:/shell/commands.rb:122 command_safe at uri:classloader:/shell/commands.rb:49 run at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427 at classpath:/jar-bootstrap.rb:223}} ``` -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-20951) Ratis LogService backed WALs
[ https://issues.apache.org/jira/browse/HBASE-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser resolved HBASE-20951. Resolution: Later > Ratis LogService backed WALs > > > Key: HBASE-20951 > URL: https://issues.apache.org/jira/browse/HBASE-20951 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Josh Elser >Priority: Major > > Umbrella issue for the Ratis+WAL work: > Design doc: > [https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit#|https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit] > The (over-simplified) goal is to re-think the current WAL APIs we have now, > ensure that they are de-coupled from the notion of being backed by HDFS, swap > the current implementations over to the new API, and then wire up the Ratis > LogService to the new WAL API. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6719) [replication] Data will lose if open a Hlog failed more than maxRetriesMultiplier
[
https://issues.apache.org/jira/browse/HBASE-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-6719.
Assignee: (was: terry zhang)
Resolution: Abandoned
> [replication] Data will lose if open a Hlog failed more than
> maxRetriesMultiplier
> -
>
> Key: HBASE-6719
> URL: https://issues.apache.org/jira/browse/HBASE-6719
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0
>Reporter: terry zhang
>Priority: Critical
> Attachments: 6719.txt, hbase-6719.patch
>
>
> Please Take a look below code
> {code:title=ReplicationSource.java|borderStyle=solid}
> protected boolean openReader(int sleepMultiplier) {
> {
> ...
> catch (IOException ioe) {
> LOG.warn(peerClusterZnode + " Got: ", ioe);
> // TODO Need a better way to determinate if a file is really gone but
> // TODO without scanning all logs dir
> if (sleepMultiplier == this.maxRetriesMultiplier) {
> LOG.warn("Waited too long for this file, considering dumping");
> return !processEndOfFile(); // Open a file failed over
> maxRetriesMultiplier(default 10)
> }
> }
> return true;
> ...
> }
> protected boolean processEndOfFile() {
> if (this.queue.size() != 0) {// Skipped this Hlog . Data loss
> this.currentPath = null;
> this.position = 0;
> return true;
> } else if (this.queueRecovered) { // Terminate Failover Replication
> source thread ,data loss
> this.manager.closeRecoveredQueue(this);
> LOG.info("Finished recovering the queue");
> this.running = false;
> return true;
> }
> return false;
> }
> {code}
> Some Time HDFS will meet some problem but actually Hlog file is OK , So after
> HDFS back ,Some data will lose and can not find them back in slave cluster.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-6712) Implement checkAndIncrement
[ https://issues.apache.org/jira/browse/HBASE-6712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6712. Assignee: (was: Ted Yu) Resolution: Later > Implement checkAndIncrement > --- > > Key: HBASE-6712 > URL: https://issues.apache.org/jira/browse/HBASE-6712 > Project: HBase > Issue Type: New Feature > Components: Client >Affects Versions: 0.92.1 >Reporter: Stefan Baritchii >Priority: Major > > increment should throw an exception if a row does not exist. instead it > creates the row. checkAndIncrement may be also a solution to it but needs > development. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6696) Add CP hooks pre and post split transaction point-of-no-return
[
https://issues.apache.org/jira/browse/HBASE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-6696.
Resolution: Won't Fix
> Add CP hooks pre and post split transaction point-of-no-return
> --
>
> Key: HBASE-6696
> URL: https://issues.apache.org/jira/browse/HBASE-6696
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 0.94.2, 0.95.2
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> In the discussion "Improving Coprocessor postSplit/postOpen synchronization"
> on user@hbase, a user implementing a secondary indexing scheme writes in:
> {quote}
> The goal with splits is to create two new daughter regions with the
> corresponding splits of the secondary indexes and lock these regions such
> that Puts/Deletes that occur while postSplit is in progress will be queued
> up so we don't run into consistency issues. [...] As of right now, the HBase
> coprocessors do not easily support a way to achieve this level of consistency
> in that there is no way to distinguish a region being opened from a split or
> a regular open.
> {quote}
> Setting aside the particulars of the use case, the issue is the {{preSplit}}
> hook does not provide the identities of the daughter regions (they don't
> exist yet) and the {{postSplit}} hook, while providing the identities of the
> daughter regions, runs after the master has processed the split and the
> daughters are already opened or opening. A CP implementer has no interception
> point available where the daughters exist but are not yet open.
> This JIRA proposes to add two new CP hooks to just before and after the
> point-of-no-return (PONR) in the split transaction: {{preSplitPONR}} and
> {{postSplitPONR}}. Perhaps the naming can be improved. This will address the
> above use case but additionally support overloading of the split transaction
> itself. We also need to address observer notification if the split
> transaction fails. This is like HBASE-5827 but scoped to this specific
> consideration only.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-6681) Propagate RegionOverloadedException to the Thrift client
[ https://issues.apache.org/jira/browse/HBASE-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6681. Resolution: Incomplete > Propagate RegionOverloadedException to the Thrift client > > > Key: HBASE-6681 > URL: https://issues.apache.org/jira/browse/HBASE-6681 > Project: HBase > Issue Type: New Feature >Reporter: Mikhail Gryzykhin >Priority: Minor > > To correctly propagate RegionOverloadedException (added in HBASE-6423) to the > Thrift client we need to make it a Thrift exception. One way to do that is to > inherit from IOError and add the desired client-side delay in milliseconds to > IOError. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6680) Allow configurable/multiple split log threads per RS.
[ https://issues.apache.org/jira/browse/HBASE-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6680. Assignee: (was: Amitanand Aiyer) Resolution: Abandoned > Allow configurable/multiple split log threads per RS. > - > > Key: HBASE-6680 > URL: https://issues.apache.org/jira/browse/HBASE-6680 > Project: HBase > Issue Type: Improvement >Reporter: Amitanand Aiyer >Priority: Minor > Attachments: HBASE-6680-94.patch > > > For rack failure case, it seems like there are multiple batches of split logs > to be processed. Allow a configurable number of split log worker per RS, so > that we split more logs quickly. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6673) Clear up the invalid ResultScanner in the ThriftServerRunner
[ https://issues.apache.org/jira/browse/HBASE-6673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6673. Resolution: Invalid > Clear up the invalid ResultScanner in the ThriftServerRunner > > > Key: HBASE-6673 > URL: https://issues.apache.org/jira/browse/HBASE-6673 > Project: HBase > Issue Type: Improvement >Reporter: Liyin Tang >Priority: Major > > Clear up the invalid ResultScanner in the ThriftServerRunner -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6676) Add build information to README.txt
[ https://issues.apache.org/jira/browse/HBASE-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6676. Resolution: Won't Fix > Add build information to README.txt > --- > > Key: HBASE-6676 > URL: https://issues.apache.org/jira/browse/HBASE-6676 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jesse Yates >Priority: Major > Labels: documentation, maven > > It would be really nice if we include build information about the current > build in the README.txt. Pulling up older versions of HBase means guessing at > the right build phase that needs to be run to build an install, since maven > can be like a crystal ball at times. To ensure DRY maybe we can roll this > into a maven build task? -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6652) [replication]replicationQueueSizeCapacity and replicationQueueNbCapacity default value is too big, Slave regionserver maybe outmemory after master start replication
[ https://issues.apache.org/jira/browse/HBASE-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6652. Assignee: (was: terry zhang) Resolution: Incomplete > [replication]replicationQueueSizeCapacity and replicationQueueNbCapacity > default value is too big, Slave regionserver maybe outmemory after master > start replication > > > Key: HBASE-6652 > URL: https://issues.apache.org/jira/browse/HBASE-6652 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.94.1 >Reporter: terry zhang >Priority: Major > > now our replication replicationQueueSizeCapacity is set to 64M and > replicationQueueNbCapacity is set to 25000. So when a master cluster with > many regionserver replicate to a small cluster 。 Slave rpc queue will full > and out of memory . > java.util.concurrent.ExecutionException: java.io.IOException: Call queue is > full, is ipc.server.max.callqueue.size too small? > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java: > 1524) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1376) > at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:700) > at > org.apache.hadoop.hbase.client.HTablePool$PooledHTable.batch(HTablePool.java:361) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.batch(ReplicationSink.java:172) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:129) > at > org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:139) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:4018) > at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:361) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1414) -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6645) Quarantine invalid reference hfiles.
[ https://issues.apache.org/jira/browse/HBASE-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6645. Resolution: Implemented > Quarantine invalid reference hfiles. > > > Key: HBASE-6645 > URL: https://issues.apache.org/jira/browse/HBASE-6645 > Project: HBase > Issue Type: New Feature >Reporter: Jonathan Hsieh >Priority: Major > > Similar to the quarantining feature introduced to hbck in HBASE-6586, we've > encountered cases of bad reference files. We should add a feature to > quarantine reference files if their parents are not present or valid. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6635) Refactor HFile version selection and exception handling.
[ https://issues.apache.org/jira/browse/HBASE-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6635. Resolution: Won't Fix > Refactor HFile version selection and exception handling. > > > Key: HBASE-6635 > URL: https://issues.apache.org/jira/browse/HBASE-6635 > Project: HBase > Issue Type: Bug >Reporter: Jonathan Hsieh >Priority: Major > Attachments: hfile.png > > > Trunk and 0.94's HFile code has some fairly convoluted code for bypassing > checksums and has mixed usage of runtime and io exceptions when error > conditions arise. This jira would clean up the code to have better > encapsulation and be more explicit about what kinds of exceptions are thrown > and what they mean. (This was partially spurred by comments in reviews of > HBASE-6586). -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6555) Avoid ssh to localhost in startup scripts
[ https://issues.apache.org/jira/browse/HBASE-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6555. Assignee: (was: Sean Busbey) Resolution: Won't Fix > Avoid ssh to localhost in startup scripts > - > > Key: HBASE-6555 > URL: https://issues.apache.org/jira/browse/HBASE-6555 > Project: HBase > Issue Type: Improvement > Components: scripts > Environment: Mac OSX Mountain Lion, HBase 89-fb >Reporter: Ramkumar Vadali >Priority: Trivial > > The use of ssh in scripts like zookeepers.sh and regionservers.sh for a > single node setup is not necessary. We can execute the command directly. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6589) RegionServer can't load class for dynamically loaded coprocessors with self defined class
[
https://issues.apache.org/jira/browse/HBASE-6589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-6589.
Resolution: Incomplete
> RegionServer can't load class for dynamically loaded coprocessors with self
> defined class
> -
>
> Key: HBASE-6589
> URL: https://issues.apache.org/jira/browse/HBASE-6589
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, regionserver
>Reporter: ShiXing
>Priority: Major
>
> When using coprocessor with custom classes like LongColumnInterpreter(mine is
> MultiColumnSchema), the coprocessor can not work for hot deploy, if the
> custom classes do not deploy in the regionserver's classpath. Although the
> self-defined class is deployed in the regions' classpath through hdfs jar.
> The exception threw at the regionserver's log:
> {code}
> 2012-08-15 16:24:24,403 ERROR org.apache.hadoop.hbase.io.HbaseObjectWritable:
> Error in readFields
> java.io.IOException: Can't find class
> com.taobao.hbase.coprocessor.MultiColumnSchema
> at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:674)
> at
> org.apache.hadoop.hbase.client.coprocessor.Exec.readFields(Exec.java:114)
> at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682)
> at
> org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:125)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1292)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1207)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:735)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:524)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:499)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.ClassNotFoundException:
> com.taobao.hbase.coprocessor.MultiColumnSchema
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
> at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.getClassByName(HbaseObjectWritable.java:784)
> at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:671)
> ... 11 more
> {code}
> It is similar as HBASE-4946, but I do not know how to solve this bug.
> If add these custom class to the RegionServer's classloader may fix it, but
> it is conflicted with HBASE-6308 to prevent dependency conflicts.
> Does anyone have some idea?
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-6600) Quarantine state for HRegions with corrupted data files
[ https://issues.apache.org/jira/browse/HBASE-6600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6600. Resolution: Incomplete > Quarantine state for HRegions with corrupted data files > --- > > Key: HBASE-6600 > URL: https://issues.apache.org/jira/browse/HBASE-6600 > Project: HBase > Issue Type: New Feature >Reporter: Jonathan Hsieh >Priority: Major > > From HBASE-6568. > Currently, corrupted HFiles cause a region to go into > RS_ZK_REGION_FAILED_OPEN, which eventually gets transitioned to > M_ZK_REGION_OFFLINE which then transitions to RS_ZK_REGION_OPENING triggering > another attempt to open the region (which fails and ...). Ideally when we > have a non-recoverable failure like corrupted hfiles, we'd transition to > QUARANTINED instead of FAILED_OPEN and stay there until an admin fixes the > problem -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6474) Separate exception types for recoverable and non-recoverable errors
[ https://issues.apache.org/jira/browse/HBASE-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6474. Resolution: Implemented > Separate exception types for recoverable and non-recoverable errors > > > Key: HBASE-6474 > URL: https://issues.apache.org/jira/browse/HBASE-6474 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Eli Collins >Priority: Major > > While reading [about > asynchbase|http://www.quora.com/HBase/What-are-the-best-HBase-client-libraries-in-each-language?srid=hqw] > I noticed that HBase turns exposes all errors to clients as IOEs rather than > use the exception hierarchy to separate recoverable and non-recoverable > exceptions. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6490) 'dfs.client.block.write.retries' value could be increased in HBase
[ https://issues.apache.org/jira/browse/HBASE-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6490. Resolution: Won't Fix > 'dfs.client.block.write.retries' value could be increased in HBase > -- > > Key: HBASE-6490 > URL: https://issues.apache.org/jira/browse/HBASE-6490 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 2.0.0 > Environment: all >Reporter: Nicolas Liochon >Priority: Minor > > When allocating a new node during writing, hdfs tries > 'dfs.client.block.write.retries' times (default 3) to write the block. When > it fails, it goes back to the nanenode for a new list, and raises an error if > the number of retries is reached. In HBase, if the error is while we're > writing a hlog file, it will trigger a region server abort (as hbase does not > trust the log anymore). For simple case (new, and as such empty log file), > this seems to be ok, and we don't lose data. There could be some complex > cases if the error occurs on a hlog file with already multiple blocks written. > Logs lines are: > "Exception in createBlockOutputStream", then "Abandoning block " followed by > "Excluding datanode " for a retry. > IOException: "Unable to create new block.", when the number of retries is > reached. > Probability of occurence seems quite low, (number of bad nodes / number of > nodes)^(number of retries), and it implies that you have a region server > without its datanode. But it's per new block. > Increasing the default value of 'dfs.client.block.write.retries' could make > sense to be better covered in chaotic conditions. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6497) Revisit HLog sizing and roll parameters
[ https://issues.apache.org/jira/browse/HBASE-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6497. Resolution: Not A Problem > Revisit HLog sizing and roll parameters > --- > > Key: HBASE-6497 > URL: https://issues.apache.org/jira/browse/HBASE-6497 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Lars George >Priority: Major > > The last major update to the HLog sizing and roll features were done in > HBASE-1394. I am proposing to revisit these settings to overcome recent > issues where the HLog becomes a major bottleneck. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6519) FSRegionScanner should be in its own file
[ https://issues.apache.org/jira/browse/HBASE-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6519. Resolution: Incomplete > FSRegionScanner should be in its own file > - > > Key: HBASE-6519 > URL: https://issues.apache.org/jira/browse/HBASE-6519 > Project: HBase > Issue Type: Improvement > Components: util > Environment: mac osx, jdk 1.6 >Reporter: Ramkumar Vadali >Priority: Minor > > I found this problem in the 0.89-fb branch. > I was not able to start the master because of a ClassNotFoundException for > FSRegionScanner. > FSRegionScanner is a top-level class in FSUtils.java. Moving it to a separate > file solved the problem. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6494) Add a delete option to Performance Evaluation
[ https://issues.apache.org/jira/browse/HBASE-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6494. Resolution: Won't Fix > Add a delete option to Performance Evaluation > - > > Key: HBASE-6494 > URL: https://issues.apache.org/jira/browse/HBASE-6494 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.89-fb, 2.0.0 >Reporter: Amitanand Aiyer >Priority: Minor > > extend performance evaluation to do deletes on the table. > will allow us to benchmark/test regression in performance for changes to the > delete path. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6526) Document FuzzyRowFilter
[ https://issues.apache.org/jira/browse/HBASE-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6526. Assignee: (was: Alex Baranau) Resolution: Incomplete > Document FuzzyRowFilter > --- > > Key: HBASE-6526 > URL: https://issues.apache.org/jira/browse/HBASE-6526 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Priority: Major > > We should document the usage of FuzzyRowFilter in HBase book / manual -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6527) Make custom filters plugin
[ https://issues.apache.org/jira/browse/HBASE-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6527. Resolution: Invalid > Make custom filters plugin > -- > > Key: HBASE-6527 > URL: https://issues.apache.org/jira/browse/HBASE-6527 > Project: HBase > Issue Type: New Feature >Reporter: Ted Yu >Priority: Major > > More and more custom Filters are created. > We should provide plugin mechanism for these custom Filters. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6453) Hbase Replication point in time feature
[ https://issues.apache.org/jira/browse/HBASE-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6453. Assignee: (was: terry zhang) Resolution: Won't Fix > Hbase Replication point in time feature > --- > > Key: HBASE-6453 > URL: https://issues.apache.org/jira/browse/HBASE-6453 > Project: HBase > Issue Type: New Feature > Components: Replication >Affects Versions: 2.0.0 >Reporter: terry zhang >Priority: Major > Attachments: hbase-6453-v1.patch > > > Now we can not control when hbase replication start to work. this patch > support we set a time stamp filter . All the row which is below this time > stamp will not be replicated. We also can delete and show this time stamp in > hbase shell if we want to change it. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6413) Investigate having hbase-env.sh decide which hadoop-compat to include
[ https://issues.apache.org/jira/browse/HBASE-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6413. Resolution: Not A Problem > Investigate having hbase-env.sh decide which hadoop-compat to include > - > > Key: HBASE-6413 > URL: https://issues.apache.org/jira/browse/HBASE-6413 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Elliott Neil Clark >Priority: Major > > Allow for one package to be created with both compat jars in and have > hbase-env load the correct one. This would simplify shipping tar.gz's -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6383) Investigate using 2Q for block cache
[ https://issues.apache.org/jira/browse/HBASE-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6383. Resolution: Won't Fix > Investigate using 2Q for block cache > > > Key: HBASE-6383 > URL: https://issues.apache.org/jira/browse/HBASE-6383 > Project: HBase > Issue Type: New Feature > Components: BlockCache, Performance, regionserver >Affects Versions: 0.95.2 >Reporter: Jesse Yates >Priority: Minor > > Currently we use a basic version of LRU to handle block caching. LRU is know > to be very susceptible to scan thrashing (not scan resistant), which is a > common operation in HBase. 2Q is an efficient caching algorithm that emulates > the effectivness of LRU/2 (eviction based not on the last access, but rather > the access before the last), but is O(1), rather than O(lg\(n)) in complexity. > JD has long been talking about investigating 2Q as it may be far better for > HBase than LRU and has been shown to be incredibly useful for traditional > database caching on production systems. > One would need to implement 2Q (though the pseudocode in the paper is quite > explicit) and then test against the existing cache implementation. > The link to the original paper is here: www.vldb.org/conf/1994/P439.PDF > A short overview of 2Q: > 2Q uses two queues (hence the name) and a list of pointers to keep track of > cached blocks. The first queue is for new, hot items (Ain). If an item is > accessed that isn't in Ain, the coldest block is evicted from Ain and the new > item replaces it. Anything accessed in Ain is already stored in memory and > kept in Ain. > When a block is evicted from Ain, it is moved to Aout _as a pointer_. If Aout > is full, the oldest element is evicted and replaced with the new pointer. > The key to 2Q comes in that when you access something in Aout, it is reloaded > into memory and stored in queue B. If B becomes full, then the coldest block > is evicted. > This essentially makes Aout a filter for long-term hot items, based on the > size of Aout. The original authors found that while you can tune Aout, it > generally performs very well at at "50% of the number of pages as would fit > into the buffer", but can be tuned as low as 5% at only a slight cost to > responsiveness to changes. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6403) RegionCoprocessorHost provides empty config when loading a coprocessor
[ https://issues.apache.org/jira/browse/HBASE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6403. Resolution: Invalid > RegionCoprocessorHost provides empty config when loading a coprocessor > -- > > Key: HBASE-6403 > URL: https://issues.apache.org/jira/browse/HBASE-6403 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Eric C. Newton >Priority: Minor > Labels: delete > > I started playing with Giraffa. I am running it against Hadoop 2.0.0-alpha, > and current HBase trunk. On line 159 of RegionCoprocessorHost, the server's > configuration is copied... or at least an attempt is made to copy it. > However, the server's configuration object, a CompoundConfiguration, does not > store the data in the same way as the base Configuration object, and so > nothing is copied. This leaves the coprocessor without access to > configuration values, like the fs.defaultFS, which Giraffa is looking for. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6344) [region_mover.rb] Look into porting updates found in Aravind G's work over into hbase.
[ https://issues.apache.org/jira/browse/HBASE-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6344. Resolution: Not A Problem > [region_mover.rb] Look into porting updates found in Aravind G's work over > into hbase. > -- > > Key: HBASE-6344 > URL: https://issues.apache.org/jira/browse/HBASE-6344 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 0.92.1, 0.94.0, 0.95.2 >Reporter: Jonathan Hsieh >Priority: Major > > Following up from HBASE-6283, we should look into porting the work from here > to hbase. > http://inchoate-clatter.blogspot.com.es/2012/03/hbase-ops-automation.html > https://github.com/aravind/hbase-utils/ > This is only relevent to 0.92+ since it has the draining node support -- 0.90 > does not have this. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6430) Few modifications in section 2.4.2.1 of Apache HBase Reference Guide
[ https://issues.apache.org/jira/browse/HBASE-6430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6430. Resolution: Not A Problem > Few modifications in section 2.4.2.1 of Apache HBase Reference Guide > > > Key: HBASE-6430 > URL: https://issues.apache.org/jira/browse/HBASE-6430 > Project: HBase > Issue Type: Improvement >Reporter: Mohammad Tariq Iqbal >Priority: Minor > Labels: beginner > Attachments: HBASE-6430.txt > > > Quite often, newbies face some issues while configuring Hbase in pseudo > distributed mode. I was no exception. I would like to propose some solutions > for these problems which worked for me. If the community finds it > appropriate, I would like to apply the patch for the same. This is the first > time I am trying to do something like this, so please pardon me if I have put > it in an appropriate manner. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6442) Add test for archiving/cleaning across multiple tables where some tables are retained and others aren't
[ https://issues.apache.org/jira/browse/HBASE-6442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6442. Resolution: Invalid > Add test for archiving/cleaning across multiple tables where some tables are > retained and others aren't > --- > > Key: HBASE-6442 > URL: https://issues.apache.org/jira/browse/HBASE-6442 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Priority: Major > > This task is continuation of previous work in > TestZooKeeperTableArchiveClient#testMultipleTables -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6402) Ability to explicitly include and exclude hosts from cluster
[ https://issues.apache.org/jira/browse/HBASE-6402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6402. Resolution: Won't Fix > Ability to explicitly include and exclude hosts from cluster > > > Key: HBASE-6402 > URL: https://issues.apache.org/jira/browse/HBASE-6402 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.94.0 >Reporter: Philip Martin >Priority: Major > > Both MR and HDFS (see > http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html; look for > "dfs.hosts", "dfs.hosts.exclude", "mapred.hosts", "mapred.hosts.exclude") > provide the user a way to deny certain slave daemons from joining the > cluster. The use for this is two-fold: it prevents developers with a client > configuration from joining the cluster from their laptop on accident, and it > provides a mechanism to explicitly decommission a node while it's in repair > or what-not. > A similar explicit cluster membership would be useful for HBase, for the same > reasons. Just yesterday a user found out that his organization was running > regionservers on machines that weren't supposed to be running regionservers. > It wasn't a huge deal, but this feature would make it easier for this user to > prevent the co-admins from making this mistake. > I'd note that HDFS and MR are inconsistent at the moment as to whether their > exclude files should have IPs, hostnames, or both. Clearly defining and > documenting that is useful. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6216) per-request tagging
[ https://issues.apache.org/jira/browse/HBASE-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6216. Assignee: (was: Li Pi) Resolution: Won't Fix > per-request tagging > --- > > Key: HBASE-6216 > URL: https://issues.apache.org/jira/browse/HBASE-6216 > Project: HBase > Issue Type: New Feature >Reporter: Kannan Muthukkaruppan >Priority: Major > > Allow applications to tag individual RPC requests so that metrics exposed by > HBase can automatically be collected/aggregated by those tags. This'll be > useful for problem diagnosis across various applications-- as it will allow > us to correlate HBase metrics (such as data block misses) back to application > level tags. > For example, > * in some multi-tenancy type use case where many different applications are > served out of a single Table or CF, it'll be useful to know the break down of > metrics (such as bytes read, data block misses, get/put RPC calls, etc.) by > application name (tag). > * even in a single app environment, the app may want to tag different API > calls with different tags even if those calls are to the same CF. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6398) Print a warning if there is no local datanode
[ https://issues.apache.org/jira/browse/HBASE-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6398. Assignee: (was: Sameer Vaishampayan) Resolution: Abandoned > Print a warning if there is no local datanode > - > > Key: HBASE-6398 > URL: https://issues.apache.org/jira/browse/HBASE-6398 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Neil Clark >Priority: Major > Labels: beginner > Attachments: 6398-4.patch > > > When starting up a RS HBase should print out a warning if there is no > datanode locally. Lots of optimizations are only available if the data is > machine local. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6218) Add dynamic on/off tracing facility to regionserver; lightweight(?) record of read/write load
[ https://issues.apache.org/jira/browse/HBASE-6218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6218. Assignee: (was: Michael Stack) Resolution: Incomplete > Add dynamic on/off tracing facility to regionserver; lightweight(?) record of > read/write load > - > > Key: HBASE-6218 > URL: https://issues.apache.org/jira/browse/HBASE-6218 > Project: HBase > Issue Type: New Feature > Components: test >Reporter: Michael Stack >Priority: Major > > It'd be sweet if we could kick a regionserver and have it start recording the > read/write load. Then after we'd taken a sample, we could turn off the > recording. > Chatting at the meetup today, replaying the WALs would give you the write > side (though missing would be the rate at which the client should play the > edits -- perhaps we could add this to the WALEdit if its not already there?). > Read side we'd need something new recording the read load (Perhaps we'd have > a single trace for read and write but somehow you could get the write from > the WAL logs). It would be nice too if we could verify that we read the > right thing somehow (hash of the return when the trace switch is thrown? > Would need to cater to differences in timestamp possibly?) -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6217) reduce overhead of maintaing get/next size metric
[
https://issues.apache.org/jira/browse/HBASE-6217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-6217.
Release Note: (was: Reduce overhead of "size metric" maintained in
StoreScanner.next().)
Assignee: (was: M. Chen)
Resolution: Abandoned
> reduce overhead of maintaing get/next size metric
> -
>
> Key: HBASE-6217
> URL: https://issues.apache.org/jira/browse/HBASE-6217
> Project: HBase
> Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Priority: Major
> Labels: patch
> Attachments: jira-6217.patch
>
>
> [Forked off this specific issue as a separate JIRA from HBASE-6066].
> Reduce overhead of "size metric" maintained in StoreScanner.next().
> {code}
> if (metric != null) {
> HRegion.incrNumericMetric(this.metricNamePrefix + metric,
>copyKv.getLength());
> }
> results.add(copyKv);
> {code}
> A single call to next() might fetch a lot of KVs. We can first add up the
> size of those KVs in a local variable and then in a finally clause increment
> the metric one shot, rather than updating AtomicLongs for each KV.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-6259) Unify all of the templates into one templating language
[ https://issues.apache.org/jira/browse/HBASE-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6259. Resolution: Won't Fix > Unify all of the templates into one templating language > --- > > Key: HBASE-6259 > URL: https://issues.apache.org/jira/browse/HBASE-6259 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.95.2 >Reporter: Elliott Neil Clark >Priority: Major > Labels: beginner > > Right now .jsp and jammon templates can't share common parts as they are > un-related. We should move to one templating language. Maybe also move away > from jamon. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6866) Timestamp consistent snapshots
[ https://issues.apache.org/jira/browse/HBASE-6866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6866. Resolution: Won't Fix > Timestamp consistent snapshots > -- > > Key: HBASE-6866 > URL: https://issues.apache.org/jira/browse/HBASE-6866 > Project: HBase > Issue Type: New Feature > Components: snapshots >Reporter: Jesse Yates >Priority: Major > > Take an timestamp-consistent snapshot of an online table. This is the one of > the two end results of the 'take a snapshot' tasks. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6360) Thrift proxy does not emit runtime metrics
[ https://issues.apache.org/jira/browse/HBASE-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6360. Assignee: (was: Michal Gregorczyk) Resolution: Incomplete > Thrift proxy does not emit runtime metrics > -- > > Key: HBASE-6360 > URL: https://issues.apache.org/jira/browse/HBASE-6360 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Karthik Ranganathan >Priority: Major > > Open jconsole against a thrift proxy, and you will not find the rumtime stats > that it should be exporting. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6180) [brainstorm] Timestamp based snapshots in HBase 0.96
[ https://issues.apache.org/jira/browse/HBASE-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6180. Resolution: Won't Fix > [brainstorm] Timestamp based snapshots in HBase 0.96 > > > Key: HBASE-6180 > URL: https://issues.apache.org/jira/browse/HBASE-6180 > Project: HBase > Issue Type: Sub-task >Reporter: Jesse Yates >Priority: Major > > Discussion ticket around doing timestamp based snapshots in HBase as an > extension/follow-on work for HBASE-6055. The implementation in HBASE-6055 (as > originally defined) is not sufficient for real-time clusters because it > requires downtime to take the snapshot. > Time-stamp based snapshots should not require downtime at the cost of > achieving global consistency. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6277) Metrics for scan object are overwritten when restart() is called
[
https://issues.apache.org/jira/browse/HBASE-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-6277.
Resolution: Invalid
> Metrics for scan object are overwritten when restart() is called
>
>
> Key: HBASE-6277
> URL: https://issues.apache.org/jira/browse/HBASE-6277
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Major
>
> From HBASE-4145:
> There's an issue in {{TableRecordReaderImpl}}. Calling restart() does this:
> {code}
> public void restart(byte[] firstRow) throws IOException {
> currentScan = new Scan(scan);
> {code}
> Which by itself is fine since the metrics will be copied from *scan* to
> *currentScan*, except that it's *currentScan* that has the updated metrics
> not *scan*.
> In other words, *currentScan* is the object that is used for scanning so it
> contains the metrics. If restart() is called, that object is overwritten by
> the original definition of the {{Scan}}. I think to fix this we could grab
> the metrics from *currentScan* first then set them back on the new object.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-6309) [MTTR] Do NN operations outside of the ZK EventThread in SplitLogManager
[
https://issues.apache.org/jira/browse/HBASE-6309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-6309.
Assignee: (was: Devaraj Das)
Resolution: Abandoned
> [MTTR] Do NN operations outside of the ZK EventThread in SplitLogManager
>
>
> Key: HBASE-6309
> URL: https://issues.apache.org/jira/browse/HBASE-6309
> Project: HBase
> Issue Type: Improvement
> Components: MTTR
>Affects Versions: 0.92.1, 0.94.0, 0.95.2
>Reporter: Jean-Daniel Cryans
>Priority: Major
>
> We found this issue during the leap second cataclysm which prompted a
> distributed splitting of all our logs.
> I saw that none of the RS were splitting after some time while the master was
> showing that it wasn't even 30% done. jstack'ing I saw this:
> {noformat}
> "main-EventThread" daemon prio=10 tid=0x7f6ce46d8800 nid=0x5376 in
> Object.wait() [0x7f6ce2ecb000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.hadoop.ipc.Client.call(Client.java:1093)
> - locked <0x0005fdd661a0> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
> at $Proxy9.rename(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy9.rename(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:759)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:253)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.moveRecoveredEditsFromTemp(HLogSplitter.java:553)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.moveRecoveredEditsFromTemp(HLogSplitter.java:519)
> at
> org.apache.hadoop.hbase.master.SplitLogManager$1.finish(SplitLogManager.java:138)
> at
> org.apache.hadoop.hbase.master.SplitLogManager.getDataSetWatchSuccess(SplitLogManager.java:431)
> at
> org.apache.hadoop.hbase.master.SplitLogManager.access$1200(SplitLogManager.java:95)
> at
> org.apache.hadoop.hbase.master.SplitLogManager$GetDataAsyncCallback.processResult(SplitLogManager.java:1011)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:571)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
> {noformat}
> We are effectively bottlenecking on doing NN operations and whatever else is
> happening in GetDataAsyncCallback. It was so bad that on our 100 offline
> cluster it took a few hours for the master to process all the incoming ZK
> events while the actual splitting took a fraction of that time.
> I'm marking this as critical and against 0.96 but depending on how involved
> the fix is we might want to backport.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-6125) Expose HBase config properties via JMX
[ https://issues.apache.org/jira/browse/HBASE-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6125. Resolution: Won't Fix > Expose HBase config properties via JMX > -- > > Key: HBASE-6125 > URL: https://issues.apache.org/jira/browse/HBASE-6125 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Otis Gospodnetic >Priority: Minor > > It would make sense to expose HBase config properties via JMX so one can > understand how HBase was configured by looking at JMX. > See: > http://search-hadoop.com/m/siI2o1rGyAj2&subj=Exposing+config+properties+via+JMX -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6232) HBase security commands should have a better error when used in insecure mode
[
https://issues.apache.org/jira/browse/HBASE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-6232.
Resolution: Incomplete
> HBase security commands should have a better error when used in insecure mode
> -
>
> Key: HBASE-6232
> URL: https://issues.apache.org/jira/browse/HBASE-6232
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 0.92.1
>Reporter: Stephen Chu
>Priority: Major
>
> I'm playing with the hbase shell's security commands and see the following.
> {noformat}
> hbase(main):002:0> grant 'schu', 'R', 'testtb', 'cf'
> ERROR: Unknown table schu!
> Here is some help for this command:
> Grant users specific rights.
> Syntax : grant
> permissions is either zero or more letters from the set "RWXCA".
> READ('R'), WRITE('W'), EXEC('X'), CREATE('C'), ADMIN('A')
> For example:
> hbase> grant 'bobsmith', 'RWXCA'
> hbase> grant 'bobsmith', 'RW', 't1', 'f1', 'col1'
> hbase(main):003:0>
> {noformat}
> {noformat}
> hbase(main):012:0> revoke 'schu', 'testtb', 'cf'
> ERROR: Unknown table schu!
> Here is some help for this command:
> Revoke a user's access rights.
> Syntax : revoke
> For example:
> hbase> revoke 'bobsmith', 't1', 'f1', 'col1'
> hbase(main):013:0>
> {noformat}
> {noformat}
> hbase(main):013:0> user_permission 'testtb'
> User Table,Family,Qualifier:Permission
>
> ERROR: Unknown table testtb!
> Here is some help for this command:
> Show all permissions for the particular user.
> Syntax : user_permission
> For example:
> hbase> user_permission
> hbase> user_permission 'table1'
> {noformat}
> Seems it just returns ERROR: Unknown table
> schu is the user, not the table.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-6254) deletes w/ many column qualifiers overwhelm Region Server
[
https://issues.apache.org/jira/browse/HBASE-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-6254.
Resolution: Incomplete
> deletes w/ many column qualifiers overwhelm Region Server
> -
>
> Key: HBASE-6254
> URL: https://issues.apache.org/jira/browse/HBASE-6254
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 0.94.0
> Environment: 5 node Cent OS + 1 master, v0.94 on cdh3u3
>Reporter: Ted Tuttle
>Priority: Major
>
> Execution of Deletes constructed with thousands of calls to
> Delete.deleteColumn(family, qualifier) are very expensive and slow.
> On our (quiet) cluster, a Delete w/ 20k qualifiers took about 13s to complete
> (as measured by client).
> When 10 such Deletes were sent to the cluster via
> HTable.delete(List), one of RegionServers ended up w/ 5 of the
> requests and became 100% CPU utilized for about 1 hour.
> This lead to the client timing out after 20min (2min x 10 retries). In one
> case, the client was able to fill the RPC callqueue and received the
> following error:
> {code}
> Failed all from region=,hostname=, port=
> java.util.concurrent.ExecutionException: java.io.IOException: Call queue is
> full, is ipc.server.max.callqueue.size too small?
> {code}
> Based on feedback (http://search-hadoop.com/m/yITsc1WcDWP), I switched to
> Delete.deleteColumn(family, qual, timestamp) where timestamp came from
> KeyValue retrieved from scan based on domain objects. This version of the
> delete ran in about 500ms.
> User group thread titled "RS unresponsive after series of deletes" has
> related logs and stacktraces.
> Link to thread: http://search-hadoop.com/m/RmIyr1WcDWP
> Here is the stack dump of region server: http://pastebin.com/8y5x4xU7
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-6105) Sweep all INFO level logging and aggressively drop to DEBUG, and from DEBUG to TRACE
[ https://issues.apache.org/jira/browse/HBASE-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6105. Resolution: Won't Fix > Sweep all INFO level logging and aggressively drop to DEBUG, and from DEBUG > to TRACE > > > Key: HBASE-6105 > URL: https://issues.apache.org/jira/browse/HBASE-6105 > Project: HBase > Issue Type: Task >Affects Versions: 0.95.2 >Reporter: Andrew Kyle Purtell >Priority: Major > > Speaking with Arjen from Facebook ops at HBaseCon, I asked if given one > single request for improving HBase operability, what would that be. The > answer was to be less verbose at INFO log level. For example, with many > regions opening, anomalous events can be difficult to pick out among the 5-6 > INFO level messages per region deployment. Where multiple INFO level messages > are printed in close succession, we should consider coalescing them. For all > INFO level messages, we should be aggressive about demoting them to DEBUG > level. And, since we are now increasing the verbosity at DEBUG level, the > same considerations should be applied there, with coalescing and demotion of > really detailed/low level logging to TRACE. > Consider making this a blocker. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6189) Snappy build instructions are out of date
[ https://issues.apache.org/jira/browse/HBASE-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6189. Fix Version/s: (was: 3.0.0-alpha-3) Assignee: (was: Sean Busbey) Resolution: Not A Problem Not a problem now that we have native hbase-compression-snappy (and hbase-compression-aircompressor) > Snappy build instructions are out of date > - > > Key: HBASE-6189 > URL: https://issues.apache.org/jira/browse/HBASE-6189 > Project: HBase > Issue Type: Bug > Components: build, documentation, Performance >Affects Versions: 0.92.1, 0.94.0, 0.95.2 >Reporter: Dave Revell >Priority: Critical > > In the ref guide (http://hbase.apache.org/book.html#build.snappy), it says to > build snappy by passing -Dsnappy. Something's wrong here, because: > 1. this causes the build to fail because the hadoop-snappy tar artifact can't > be found by maven > 2. the snappy classes are already included in hadoop 1.0, so using the snappy > profile is unnecessary > It would be great if someone who knows when/why to use the snappy profile > could fix the instructions (and fix the POM if necessary). -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6194) add open time for a region and list recently closed regions in a regionserver UI
[ https://issues.apache.org/jira/browse/HBASE-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6194. Resolution: Not A Problem > add open time for a region and list recently closed regions in a regionserver > UI > > > Key: HBASE-6194 > URL: https://issues.apache.org/jira/browse/HBASE-6194 > Project: HBase > Issue Type: Improvement >Reporter: Feifei Ji >Priority: Major > > The region server currently lists all the regions that it is hosting. It will > be useful to report when those regions were opened on this server. It will > also be useful to report what and when were the recent regions closed. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6136) Optimize the multi get to be executed in parallel.
[ https://issues.apache.org/jira/browse/HBASE-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6136. Release Note: (was: Optimize the multi get to be executed in parallel using multi thread in hbase client ) Resolution: Won't Fix > Optimize the multi get to be executed in parallel. > -- > > Key: HBASE-6136 > URL: https://issues.apache.org/jira/browse/HBASE-6136 > Project: HBase > Issue Type: Improvement >Reporter: Liyin Tang >Priority: Major > Attachments: HBASE-6136-trunk.patch > > > Currently the multiAction will be executed by one IPC handler thread in the > server side. It would be more efficient to optimize the multi get > particularly to be executed in parallel since the multiget doesn't need to > write to the WAL. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6143) Make region assignment smarter when regions are re-enabled.
[ https://issues.apache.org/jira/browse/HBASE-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6143. Resolution: Abandoned > Make region assignment smarter when regions are re-enabled. > --- > > Key: HBASE-6143 > URL: https://issues.apache.org/jira/browse/HBASE-6143 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Elliott Neil Clark >Priority: Critical > Labels: balancer > Attachments: 6143-v1.txt, 6143-v2.txt, 6143-v3.txt, > HBASE-6143-0.patch, HBASE-6143-1.patch, HBASE-6143-v1.patch, > HBASE-6143-v2.patch > > > Right now a random region server is picked when re-enabling a table. This > could be much smarter. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6137) RegionServer-level context and start/stop life-cycle methods for observer coprocessor
[ https://issues.apache.org/jira/browse/HBASE-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6137. Resolution: Won't Fix > RegionServer-level context and start/stop life-cycle methods for observer > coprocessor > - > > Key: HBASE-6137 > URL: https://issues.apache.org/jira/browse/HBASE-6137 > Project: HBase > Issue Type: New Feature > Components: Coprocessors >Affects Versions: 2.0.0 >Reporter: James Taylor >Priority: Major > > Coprocessors are a great way for an application to affect server-side > processing. We're using observer coprocessors via the postScannerOpen to > enable a scan to do aggregation. There's currently no way, however, to > store/share state across coprocessor invocations on the regions within a > region server. Ideally, we'd like to be able to have a context object that > allows state to be shared across coprocessor invocation for the regions on > the same region server. This would save us the setup cost for "compiling" our > aggregators again for each region. Also useful, would be: > - a start/stop method invocation on this new region server context object > before the first region invocation and after the last region invocation on a > given region server. > - a way to pass state to the start/stop method from the client. The > scan.setAttribute works well for passing state for the invocation on each > region, but ideally something that would allow state to be passed just once > per region server. One use case would be to pass a cache of the row data for > a hash join implementation, where we wouldn't want to pass this information > for every region. > Our current work around is to either take the hit of the extra setup costs > for the coprocessor invocation on each region or use an Endpoint coprocessor > to initialize state prior to the client scan that will cause coprocessor > invocations. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5969) HRI.getRegionName/AsString are inconsistent for regions with the old encoding scheme
[ https://issues.apache.org/jira/browse/HBASE-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5969. Resolution: Not A Problem > HRI.getRegionName/AsString are inconsistent for regions with the old encoding > scheme > > > Key: HBASE-5969 > URL: https://issues.apache.org/jira/browse/HBASE-5969 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Jean-Daniel Cryans >Priority: Major > > HRI.getRegionName and getRegionNameAsString don't give consistent results for > regions created with the old encoding scheme. See Aravind's comment in > HBASE-5929 on how a region was appended with ".1290583321" when its HRI was > queried with getRegionNameAsString and, once passed to HBA, wasn't able to > compact it (I checked .META. and it doesn't have that last part so it must > come from getRegionName). > This one might be a little hard to fix if we're already dependent on the > broken behavior of getRegionName. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5960) Expose HBase HDFS disk space usage via JMX
[ https://issues.apache.org/jira/browse/HBASE-5960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5960. Resolution: Won't Fix > Expose HBase HDFS disk space usage via JMX > -- > > Key: HBASE-5960 > URL: https://issues.apache.org/jira/browse/HBASE-5960 > Project: HBase > Issue Type: Improvement > Components: metrics, monitoring >Reporter: Otis Gospodnetic >Priority: Minor > > HBase should expose via JMX how much HDFS disk space it is using. See > http://search-hadoop.com/m/s9VIx4Hhjz . -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5860) splitlogmanager should not unnecessarily resubmit tasks when zk unavailable
[ https://issues.apache.org/jira/browse/HBASE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5860. Assignee: (was: Prakash Khemani) Resolution: Abandoned > splitlogmanager should not unnecessarily resubmit tasks when zk unavailable > --- > > Key: HBASE-5860 > URL: https://issues.apache.org/jira/browse/HBASE-5860 > Project: HBase > Issue Type: Improvement >Reporter: Prakash Khemani >Priority: Major > Attachments: > 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch, > 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch > > > (Doesn't really impact the run time or correctness of log splitting) > say the master has lost connection to zk. splitlogmanager's timeoutmanager > will realize that all the tasks that were submitted are still unassigned. It > will resubmit those tasks (i.e. create dummy znodes) > splitlogmanager should realze that the tasks are unassigned but their znodes > have not been created. > 012-04-20 13:11:20,516 INFO org.apache.hadoop.hbase.master.SplitLogManager: > dead splitlog worker msgstore295.snc4.facebook.com,60020,1334948757026 > 2012-04-20 13:11:20,517 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: > Scheduling batch of logs to split > 2012-04-20 13:11:20,517 INFO org.apache.hadoop.hbase.master.SplitLogManager: > started splitting logs in > [hdfs://msgstore215.snc4.facebook.com:9000/MSGSTORE215-SNC4-HBASE/.logs/msgstore295.snc4.facebook.com,60020,1334948757026-splitting] > 2012-04-20 13:11:20,565 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server msgstore235.snc4.facebook.com/10.30.222.186:2181 > 2012-04-20 13:11:20,566 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to msgstore235.snc4.facebook.com/10.30.222.186:2181, > initiating session > 2012-04-20 13:11:20,575 INFO org.apache.hadoop.hbase.master.SplitLogManager: > total tasks = 4 unassigned = 4 > 2012-04-20 13:11:20,576 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: > resubmitting unassigned task(s) after timeout > 2012-04-20 13:11:21,577 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: > resubmitting unassigned task(s) after timeout > 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x36ccb0f8010002, likely server has > closed socket, closing socket connection and attempting reconnect > 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read > additional data from server sessionid 0x136ccb0f489, likely server has > closed socket, closing socket connection and attempting reconnect > 2012-04-20 13:11:21,786 WARN > org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc > =CONNECTIONLOSS for > /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951586677 > retry=3 > 2012-04-20 13:11:21,786 WARN > org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc > =CONNECTIONLOSS for > /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951920332 > retry=3 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5979) Non-pread DFSInputStreams should be associated with scanners, not HFile.Readers
[ https://issues.apache.org/jira/browse/HBASE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5979. Resolution: Incomplete > Non-pread DFSInputStreams should be associated with scanners, not > HFile.Readers > --- > > Key: HBASE-5979 > URL: https://issues.apache.org/jira/browse/HBASE-5979 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver >Reporter: Todd Lipcon >Priority: Critical > > Currently, every HFile.Reader has a single DFSInputStream, which it uses to > service all gets and scans. For gets, we use the positional read API (aka > "pread") and for scans we use a synchronized block to seek, then read. The > advantage of pread is that it doesn't hold any locks, so multiple gets can > proceed at the same time. The advantage of seek+read for scans is that the > datanode starts to send the entire rest of the HDFS block, rather than just > the single hfile block necessary. So, in a single thread, pread is faster for > gets, and seek+read is faster for scans since you get a strong pipelining > effect. > However, in a multi-threaded case where there are multiple scans (including > scans which are actually part of compactions), the seek+read strategy falls > apart, since only one scanner may be reading at a time. Additionally, a large > amount of wasted IO is generated on the datanode side, and we get none of the > earlier-mentioned advantages. > In one test, I switched scans to always use pread, and saw a 5x improvement > in throughput of the YCSB scan-only workload, since it previously was > completely blocked by contention on the DFSIS lock. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5994) Document and add to TableMapReduceUtil setup if security is enabled
[
https://issues.apache.org/jira/browse/HBASE-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-5994.
Resolution: Won't Fix
> Document and add to TableMapReduceUtil setup if security is enabled
> ---
>
> Key: HBASE-5994
> URL: https://issues.apache.org/jira/browse/HBASE-5994
> Project: HBase
> Issue Type: Improvement
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> If secure RPC is enabled, then users must enable the TokenProvider
> coprocessor. They also must acquire a job token and add it to the job
> configuration, by way of something like:
> {code}
> if (User.isHBaseSecurityEnabled()) try {
> User.getCurrent().obtainAuthTokenForJob(conf, job);
> } catch (InterruptedException ie) {
> Thread.interrupted();
> }
> {code}
> TableMapReduceUtil will do this automatically.
> Add some detail about this to the manual.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-5815) ZKUtil.createWithParents should be transactional
[ https://issues.apache.org/jira/browse/HBASE-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5815. Resolution: Abandoned > ZKUtil.createWithParents should be transactional > > > Key: HBASE-5815 > URL: https://issues.apache.org/jira/browse/HBASE-5815 > Project: HBase > Issue Type: Improvement >Reporter: Jesse Yates >Priority: Major > Labels: zookeeper > Attachments: java_HBASE-5815-v0.patch > > > Should solve a lot of tricky corner cases when you create the parent, get an > update to watchers (who read ZK, but find no children) and then create the > children. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5947) Check for valid user/table/family/qualifier and acl state
[ https://issues.apache.org/jira/browse/HBASE-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5947. Resolution: Incomplete > Check for valid user/table/family/qualifier and acl state > - > > Key: HBASE-5947 > URL: https://issues.apache.org/jira/browse/HBASE-5947 > Project: HBase > Issue Type: Improvement > Components: security >Affects Versions: 0.92.1, 0.94.0, 0.95.2 >Reporter: Matteo Bertozzi >Priority: Major > Labels: acl > > HBase Shell grant/revoke doesn't check for valid user or > table/family/qualifier so can you end up having rights for something that > doesn't exists. > We might also want to ensure, upon table/column creation, that no entries are > already stored at the acl table. We might still have residual acl entries if > something goes wrong, in postDeleteTable(), postDeleteColumn(). -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5950) Add a decimal comparator for Filter
[ https://issues.apache.org/jira/browse/HBASE-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5950. Assignee: (was: Jieshan Bean) Resolution: Incomplete > Add a decimal comparator for Filter > --- > > Key: HBASE-5950 > URL: https://issues.apache.org/jira/browse/HBASE-5950 > Project: HBase > Issue Type: New Feature > Components: Filters >Affects Versions: 0.94.0, 0.95.2 >Reporter: Jieshan Bean >Priority: Major > > Suppose we have a requirement like below: > we want to get the rows with one specified column value larger than A and > less than B. > (They are all decimals or integers) > namely: >A < Integer.valueof(column) < B > Use BinaryComparator will not help us to archive that goal: > e.g. suppose A = 100, B = 200, one column value is 11. >So it can satisfy that condition, but it's not the row we wanted. > > So I suggest to add one comparator to help implementing this. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5761) [Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the docs suggest otherwise
[
https://issues.apache.org/jira/browse/HBASE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-5761.
Resolution: Incomplete
> [Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the
> docs suggest otherwise
> ---
>
> Key: HBASE-5761
> URL: https://issues.apache.org/jira/browse/HBASE-5761
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Wouter Bolsterlee
>Priority: Trivial
> Labels: beginner
>
> It seems to me there is an inconsistency (or error) in the Thrift2
> {{TDelete}} struct and its documentation. The docs for the {{TDelete}} struct
> state:
> {quote}
> If no timestamp is specified the most recent version will be deleted. To
> delete all previous versions, specify the DELETE_COLUMNS TDeleteType.
> {quote}
> ...which implies that the default is {{TDeleteType.DELETE_COLUMN}}
> (singular), not {{TDeleteType.DELETE_COLUMNS}} (plural).
> However, the {{deleteType}} field in the {{TDelete}} struct defaults to the
> value {{1}}, which is {{TDeleteType.DELETE_COLUMNS}} (plural) in
> {{/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift}}. The
> field is currently (r1239241) defined as follows:
> {{4: optional TDeleteType deleteType = 1,}}
> I'd suggest that the default for this optional field is changed to
> {{TDeleteType.DELETE_COLUMN}} (singular). The line above from the {{TDelete}}
> struct would then become:
> {{4: optional TDeleteType deleteType = 0,}}
> Since this change just involves changing a {{1}} into a {{0}}, I'll leave the
> trivial patch to someone who can also commit it in one go. Thanks in advance.
> :)
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-5796) Fix our abuse of IOE: see http://blog.tsunanet.net/2012/04/apache-hadoop-abuse-ioexception.html
[ https://issues.apache.org/jira/browse/HBASE-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5796. Resolution: Not A Problem > Fix our abuse of IOE: see > http://blog.tsunanet.net/2012/04/apache-hadoop-abuse-ioexception.html > --- > > Key: HBASE-5796 > URL: https://issues.apache.org/jira/browse/HBASE-5796 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Priority: Major > > Lets make more context particular exceptions rather than throw IOEs > everywhere. See Benoît's rant: > http://blog.tsunanet.net/2012/04/apache-hadoop-abuse-ioexception.html -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5786) Implement histogram metrics for flush and compaction latencies and sizes.
[ https://issues.apache.org/jira/browse/HBASE-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5786. Resolution: Won't Fix > Implement histogram metrics for flush and compaction latencies and sizes. > - > > Key: HBASE-5786 > URL: https://issues.apache.org/jira/browse/HBASE-5786 > Project: HBase > Issue Type: New Feature > Components: metrics, regionserver >Affects Versions: 0.92.2, 0.94.0, 0.95.2 >Reporter: Jonathan Hsieh >Priority: Major > > Average time for region operations doesn't really tell a useful story when > that help diagnose anomalous conditions. > It would be extremely useful to add histogramming metrics similar to > HBASE-5533 for region operations like flush, compaction and splitting. The > probably should be forward biased at a much coarser granularity however > (maybe decay every day?) -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5839) Backup master not starting up due to Bind Exception while starting HttpServer
[ https://issues.apache.org/jira/browse/HBASE-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5839. Resolution: Incomplete > Backup master not starting up due to Bind Exception while starting HttpServer > - > > Key: HBASE-5839 > URL: https://issues.apache.org/jira/browse/HBASE-5839 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Priority: Major > > Backup master tries to bind to the info port 60010. > This is done once the back up master becomes active. Even before that the > Data Xceviers threads (IPC handlers) are started and they are started at > random port. If already 60010 is used then when standby master comes up then > it fails due to bind exception. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5713) Introduce throttling during Instant schema change process to throttle opening/closing regions.
[ https://issues.apache.org/jira/browse/HBASE-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5713. Assignee: (was: Subbu M Iyer) Resolution: Abandoned > Introduce throttling during Instant schema change process to throttle > opening/closing regions. > --- > > Key: HBASE-5713 > URL: https://issues.apache.org/jira/browse/HBASE-5713 > Project: HBase > Issue Type: Bug > Components: Client, master, regionserver, shell >Reporter: Subbu M Iyer >Priority: Minor > Attachments: 5713.txt, patch-v4.patch > > > There is a potential for region open/close stampede during instant schema > change process as the process attempts to close/open impacted regions in > rapid succession. We need to introduce some kind of throttling to eliminate > the race condition. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5716) Make HBASE-4608 easier to use
[
https://issues.apache.org/jira/browse/HBASE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-5716.
Assignee: (was: Li Pi)
Resolution: Incomplete
> Make HBASE-4608 easier to use
> -
>
> Key: HBASE-5716
> URL: https://issues.apache.org/jira/browse/HBASE-5716
> Project: HBase
> Issue Type: Improvement
>Reporter: Jean-Daniel Cryans
>Priority: Major
>
> HBASE-4608 is a nice feature but after playing with it for a while I think
> the following should be fixed to make it easier to use by someone who's not a
> dev:
> - Add some signal that says that the feature is turned on. Right now you can
> {{jstack | grep KeyValueCompression}} a couple of times and if you get a hit
> you definitely know it's on, but otherwise the random user wouldn't know
> without going through the jira.
> - Add documentation in the reference guide. At the minimum add
> {{hbase.regionserver.wal.enablecompression}} in there with a small
> description. Better would be to add a section in {{Appendix B}} or something
> like that and describe the functionality a bit and who it's useful for. For
> example, flush from your brain the knowledge of the patch and read the name
> of the configuration... now let's say you have a use case that involves
> writing easily compressible values. Any normal user would believe that this
> is a good tuning parameter for them, but it's just going to waste CPU cycles.
> - Add some metrics like we have for HFiles where you get a clue about the
> compression ratio.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-5673) The OOM problem of IPC client call cause all handle block
[ https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5673. Hadoop Flags: (was: Incompatible change) Assignee: (was: xufeng) Resolution: Abandoned > The OOM problem of IPC client call cause all handle block > -- > > Key: HBASE-5673 > URL: https://issues.apache.org/jira/browse/HBASE-5673 > Project: HBase > Issue Type: Bug > Environment: 0.90.6 >Reporter: xufeng >Priority: Major > Attachments: HBASE-5673-90-V2.patch, HBASE-5673-90.patch > > > if HBaseClient meet "unable to create new native thread" exception, the call > will never complete because it be lost in calls queue. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5626) Compactions simulator tool for proofing algorithms
[ https://issues.apache.org/jira/browse/HBASE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5626. Assignee: (was: Sreeram Venkatasubramanian) Resolution: Won't Fix > Compactions simulator tool for proofing algorithms > -- > > Key: HBASE-5626 > URL: https://issues.apache.org/jira/browse/HBASE-5626 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Priority: Minor > Labels: beginner > Attachments: cf_compact.py > > > A tool to run compaction simulations would be a nice to have. We could use > it to see how well an algo ran under different circumstances loaded w/ > different value types with different rates of flushes and splits, etc. > HBASE-2462 had one (see in patch). Or we could try doing it using something > like this: http://en.wikipedia.org/wiki/Discrete_event_simulation -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5612) Data types for HBase values
[ https://issues.apache.org/jira/browse/HBASE-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5612. Assignee: (was: Mikhail Gryzykhin) Resolution: Incomplete > Data types for HBase values > --- > > Key: HBASE-5612 > URL: https://issues.apache.org/jira/browse/HBASE-5612 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Gryzykhin >Priority: Major > > In many real-life applications all values in a certain column family are of a > certain data type, e.g. 64-bit integer. We could specify that in the column > descriptor and enable data type-specific compression such as variable-length > integer encoding. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5622) Improve efficiency of mapred vesion of RowCounter
[ https://issues.apache.org/jira/browse/HBASE-5622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5622. Resolution: Won't Fix > Improve efficiency of mapred vesion of RowCounter > - > > Key: HBASE-5622 > URL: https://issues.apache.org/jira/browse/HBASE-5622 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Priority: Minor > Attachments: HBASE-5622.patch > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5600) Make Endpoint Coprocessors Available from Thrift
[ https://issues.apache.org/jira/browse/HBASE-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5600. Resolution: Won't Fix > Make Endpoint Coprocessors Available from Thrift > > > Key: HBASE-5600 > URL: https://issues.apache.org/jira/browse/HBASE-5600 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Ben West >Priority: Minor > Labels: thrift, thrift2 > > Currently, the only way to access an endpoint coprocessor via thrift is to > modify the schema and Thrift server for every coprocessor function. This is > annoying. It should be possible to use your coprocessors without having to > mangle HBase core code (since that's the point of coprocessors). -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-5580) Publish Thrift-generated files for other languages
[ https://issues.apache.org/jira/browse/HBASE-5580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-5580. Resolution: Won't Fix > Publish Thrift-generated files for other languages > -- > > Key: HBASE-5580 > URL: https://issues.apache.org/jira/browse/HBASE-5580 > Project: HBase > Issue Type: New Feature > Components: Thrift >Affects Versions: 0.90.4 >Reporter: Patrick Angeles >Priority: Major > Labels: thrift, thrift2 > > HBase ships with Thrift-generated Java files for use with the ThriftServer. > For convenience (and to save users the frustration of having to compile and > install the Thrift compiler), HBase can ship with the thrift-generated files > for other languages as well. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-10494) hadoop2 class reference in Maven Central's hbase-client-0.96.1.1-hadoop1
[
https://issues.apache.org/jira/browse/HBASE-10494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-10494.
-
Resolution: Incomplete
> hadoop2 class reference in Maven Central's hbase-client-0.96.1.1-hadoop1
>
>
> Key: HBASE-10494
> URL: https://issues.apache.org/jira/browse/HBASE-10494
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.1.1
> Environment: Only affects jar on Maven Central. Jar in the hadoop1
> tarball download is not affected.
>Reporter: Dan LaRocque
>Priority: Minor
>
> RpcClient$Connection.class as shipped in the hbase-client-0.96.1.1-hadoop1
> jar on Maven Central contains references to
> org.apache.hadoop.net.SocketInputWrapper. I think this class does not exist
> in hadoop1 because a classfile search of central yields hits only on 2.0 and
> 0.23. There may be other references. I only know about this one because it
> was killing my HRegionServer early with this exception:
> {noformat}
> 2014-02-10 20:55:52,021 INFO [M:0;dalarolap:48768] master.ServerManager:
> Waiting for region servers count to settle; currently checked in 0, slept for
> 0 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms,
> interval of 1500 ms.
> 2014-02-10 20:55:52,066 WARN [RS:0;dalarolap:33703]
> regionserver.HRegionServer: error telling master we are up
> com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError:
> org/apache/hadoop/net/SocketInputWrapper
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1670)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1711)
> at
> org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:5402)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1926)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:784)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/net/SocketInputWrapper
> at
> org.apache.hadoop.hbase.ipc.RpcClient.createConnection(RpcClient.java:348)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1522)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1424)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1653)
> ... 5 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.net.SocketInputWrapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 9 more
> {noformat}
> I first stumbled over this while developing an app managed by Maven that
> depends on hbase-client, but then reproduced it by extracting the hadoop1
> tarball and replacing the client jar with the same-named one from Maven
> Central.
> I think this is not the same as HBASE-7269, although the stacktrace is
> similar.
> Here's a disassembler grep on the Maven Central copy showing some references:
> {noformat}
> # From the root of an extracted hbase-client-0.96.1.1-hadoop1 jar downloaded
> from Maven Central
> client-maven$ javap -verbose
> 'org/apache/hadoop/hbase/ipc/RpcClient$Connection.class' | grep
> SocketInputWrapper
>#198 = Methodref #744.#818//
> org/apache/hadoop/net/NetUtils.getInputStream:(Ljava/net/Socket;)Lorg/apache/hadoop/net/SocketInputWrapper;
>#818 = NameAndType#1115:#1159 //
> getInputStream:(Ljava/net/Socket;)Lorg/apache/hadoop/net/SocketInputWrapper;
> #1159 = Utf8
> (Ljava/net/Socket;)Lorg/apache/hadoop/net/SocketInputWrapper;
>180: invokestatic #198// Method
> org/apache/hadoop/net/NetUtils.getInputStream:(Ljava/net/Socket;)Lorg/apache/hadoop/net/SocketInputWrapper;
> {noformat}
> Here's the same grep on the tarball's copy. No references.
> {noformat}
> # Same as above, but using jar from the download tarball for hadoop1
> client-tarball$ javap -verbose
> 'org/apache/hadoop/hbase/ipc/RpcClient$Connection.class' | grep
> SocketInputWrapper
> client-tarball$
> {noformat}
> What do you think?
--
This message was sent by Atlassian Jira
(v8.
[jira] [Resolved] (HBASE-10554) Please delete old releases from mirroring system
[ https://issues.apache.org/jira/browse/HBASE-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-10554. - Resolution: Fixed > Please delete old releases from mirroring system > > > Key: HBASE-10554 > URL: https://issues.apache.org/jira/browse/HBASE-10554 > Project: HBase > Issue Type: Task >Affects Versions: 0.96.0, 0.96.1, 0.94.14, 0.94.15 >Reporter: Sebb >Priority: Major > > To reduce the load on the ASF mirrors, projects are required to delete old > releases [1] > Please can you remove all non-current releases? > Thanks! > [1] http://www.apache.org/dev/release.html#when-to-archive -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-10623) Add ability to read a row after applying a RowMutation
[ https://issues.apache.org/jira/browse/HBASE-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-10623. - Resolution: Won't Fix > Add ability to read a row after applying a RowMutation > -- > > Key: HBASE-10623 > URL: https://issues.apache.org/jira/browse/HBASE-10623 > Project: HBase > Issue Type: Improvement >Reporter: Michael Webster >Priority: Minor > > It would be useful to be able to return a row to the user after all of the > mutations in a RowMutation operation has been applied. This would be similar > to transactions in Redis where the result of executing an operation is > returned to the user. However, since RowMutations only affect a single row, > only the final result would need to be returned. This could allow for a > snapshot view of the row without requiring any timestamp manipulation, or a > second read by the client. > Looking at the implementation of RowMutations, it seems like the Get could be > done after the write has been committed but before the row locks are released. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-6193) REST Api for HBase administrative tasks
[ https://issues.apache.org/jira/browse/HBASE-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-6193. Resolution: Won't Fix > REST Api for HBase administrative tasks > --- > > Key: HBASE-6193 > URL: https://issues.apache.org/jira/browse/HBASE-6193 > Project: HBase > Issue Type: New Feature >Reporter: Eric Yang >Priority: Major > > For enabling HBASE-4368, it may be useful to have embedded REST API server > for each region server, and HBase Master can either aggregate or > proxy/redirect REST API base on administrative task functions. > The popular admin task that can take advantage of this setup are: > - Trigger table compaction > - Show process list > - Health check > - Show black listed servers -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-9365) Optionally limit major compactions to off peak hours
[ https://issues.apache.org/jira/browse/HBASE-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-9365. Resolution: Won't Fix > Optionally limit major compactions to off peak hours > > > Key: HBASE-9365 > URL: https://issues.apache.org/jira/browse/HBASE-9365 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Priority: Major > > We already have off peak hours (where we set a more aggressive compaction > ratio) and periodic major compactions. > It would be nice if we could limit this to off peak hours as well. > A major compaction can be triggered in three ways: > # a periodic chore checking every ~3h (10.000.000ms) by default > # a minor compaction promoted to major because the last major compaction was > too long ago > # a minor compaction promoted to major (at the HStore level), because it > touched all HFiles anyway > For case #1 and #2 we could could optionally return false from > Store.isMajorCompaction(...) when we're not in the off peak window. Would > have a new config option to enforce that only optionally. > In case #3 we're compacting all files anyway, so not need to interfere with > that. > Thoughts? -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-9588) Expose checkAndPut/checkAndDelete with comparators to HTableInterface
[ https://issues.apache.org/jira/browse/HBASE-9588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-9588. Resolution: Abandoned > Expose checkAndPut/checkAndDelete with comparators to HTableInterface > - > > Key: HBASE-9588 > URL: https://issues.apache.org/jira/browse/HBASE-9588 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Robert Roland >Priority: Major > Attachments: checkAndPut_HBASE-9588_0.94.patch, > checkAndPut_HBASE-9588_TRUNK-v2.patch, checkAndPut_HBASE-9588_TRUNK-v3.patch, > checkAndPut_HBASE-9588_TRUNK.patch, checkAndPut_HBASE-9588_TRUNK.patch.1 > > > HRegionInterface allows you to specify a comparator to checkAndPut and > checkAndDelete, but that isn't available to the standard HTableInterface. > The attached patches expose these functions to the client. It adds two > methods to HTableInterface, which required implementing in several places. > They are not implemented in RemoteHTable - I couldn't see an obvious way to > implement there. Following the pattern of increment, batch, etc, they are > "not supported." -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-7582) Unit test execution mode that randomizes test order
[ https://issues.apache.org/jira/browse/HBASE-7582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-7582. Resolution: Won't Fix > Unit test execution mode that randomizes test order > --- > > Key: HBASE-7582 > URL: https://issues.apache.org/jira/browse/HBASE-7582 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Andrew Kyle Purtell >Priority: Minor > > I've also come around to agreeing with Jon Hsieh that we should have a test > mode that somehow randomizes test order to catch issues where unit test cases > are written with the assumption of a certain order. It should not be on by > default but can be enabled when evaluating patches for commit that include > new tests. > Case in point is HBASE-7581, which seems to be due to a new test added in > HBASE-5498. The patch on HBASE-5498 tested out all green in local tests prior > to commit. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-9291) Enable client to setAttribute that is sent once to each region server
[ https://issues.apache.org/jira/browse/HBASE-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-9291. Resolution: Won't Fix > Enable client to setAttribute that is sent once to each region server > - > > Key: HBASE-9291 > URL: https://issues.apache.org/jira/browse/HBASE-9291 > Project: HBase > Issue Type: New Feature > Components: IPC/RPC >Reporter: James Taylor >Priority: Major > > Currently a Scan and Mutation allow the client to set its own attributes that > get passed through the RPC layer and are accessible from a coprocessor. This > is very handy, but breaks down if the amount of information is large, since > this information ends up being sent again and again to every region. Clients > can work around this with an endpoint "pre" and "post" coprocessor invocation > that: > 1) sends the information and caches it on the region server in the "pre" > invocation > 2) invokes the Scan or sends the batch of Mutations, and then > 3) removes it in the "post" invocation. > In this case, the client is forced to identify all region servers (ideally, > all region servers that will be involved in the Scan/Mutation), make extra > RPC calls, manage the caching of the information on the region server, > age-out the information (in case the client dies before step (3) that clears > the cached information), and must deal with the possibility of a split > occurring while this operation is in-progress. > Instead, it'd be much better if an attribute could be identified as a "region > server" attribute in OperationWithAttributes and the HBase RPC layer would > take care of doing the above. > The use case where the above are necessary in Phoenix include: > 1) Hash joins, where the results of the smaller side of a join scan are > packaged up and sent to each region server, and > 2) Secondary indexing, where the metadata of knowing a) which column > family/column qualifier pairs and b) which part of the row key contributes to > which indexes are sent to each region server that will process a batched put. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-9635) HBase Table regions are not getting re-assigned to the new region server when it comes up (when the existing region server not able to handle the load)
[
https://issues.apache.org/jira/browse/HBASE-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-9635.
Resolution: Incomplete
> HBase Table regions are not getting re-assigned to the new region server when
> it comes up (when the existing region server not able to handle the load)
>
>
> Key: HBASE-9635
> URL: https://issues.apache.org/jira/browse/HBASE-9635
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Affects Versions: 0.94.11
> Environment: SuSE11
>Reporter: shankarlingayya
>Priority: Major
>
> {noformat}
> HBase Table regions are not getting assigned to the new region server for a
> period of 30 minutes (when the existing region server not able to handle the
> load)
> Procedure:
> 1. Setup Non HA Hadoop Cluster with two nodes (Node1-XX.XX.XX.XX,
> Node2-YY.YY.YY.YY)
> 2. Install Zookeeper & HRegionServer in Node-1
> 3. Install HMaster & HRegionServer in Node-2
> 4. From Node2 create HBase Table ( table name 't1' with one column family
> 'cf1' )
> 5. Perform addrecord 99649 rows
> 6. kill both the node Region Server and limit the Node1 Region Server FD to
> 600
> 7. Start only the Node1 Region server ==> so that FD exhaust can happen in
> Node1 Region Server
> 8. After some 5-10 minuites start the Node2 Region Server
> ===> Huge number of regions of table 't1' are in OPENING state, which are not
> getting re assigned to the Node2 region server which is free.
> ===> When the new region server comes up then the master should detect and
> allocate the open failed regions to the region server (here it is staying the
> OPENINING state for 30 minutes which will have huge impcat user app which
> makes use of this table)
> 2013-09-23 18:46:12,160 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> Instantiated t1,row507465,1379937224590.2d9fad2aee78103f928d8c7fe16ba6cd.
> 2013-09-23 18:46:12,160 ERROR
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
> of region=t1,row507465,1379937224590.2d9fad2aee78103f928d8c7fe16ba6cd.,
> starting to roll back the global memstore size.
> 2013-09-23 18:50:55,284 WARN org.apache.hadoop.hdfs.LeaseRenewer: Failed to
> renew lease for
> [DFSClient_hb_rs_HOST-XX.XX.XX.XX,61020,1379940823286_-641204614_48] for 309
> seconds. Will retry shortly ...
> java.io.IOException: Failed on local exception: java.net.SocketException: Too
> many open files; Host Details : local host is:
> "HOST-XX.XX.XX.XX/XX.XX.XX.XX"; destination host is: "HOST-XX.XX.XX.XX":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
> at org.apache.hadoop.ipc.Client.call(Client.java:1351)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at $Proxy13.renewLease(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:188)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at $Proxy13.renewLease(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:522)
> at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:679)
> at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
> at
> org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.SocketException: Too many open files
> at sun.nio.ch.Net.socket0(Native Method)
> at sun.nio.ch.Net.socket(Net.java:97)
> at sun.nio.ch.SocketChannelImpl.(SocketChannelImpl.java:84)
> at
> sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:37)
> at java.nio.channels.SocketChannel.open(SocketChannel.java:105)
> at
> org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:62)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:523)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
> a
[jira] [Resolved] (HBASE-9641) We should have a way to provide table level based ACL.
[ https://issues.apache.org/jira/browse/HBASE-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-9641. Resolution: Incomplete > We should have a way to provide table level based ACL. > -- > > Key: HBASE-9641 > URL: https://issues.apache.org/jira/browse/HBASE-9641 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Jean-Marc Spaggiari >Priority: Minor > > Today we can grant rights to users based on the user / table / column family > / family. When there is thousands of users and you want to add a new table, > it's long to add back everyone to the table. > We should be able to provide a table based ACL. Something like "grant_table > [ [ ]]" to give specific > rights to a table for ALL the users. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-24527) Improve region housekeeping status observability
[
https://issues.apache.org/jira/browse/HBASE-24527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-24527.
-
Resolution: Implemented
I ended up addressing this simply with the new shell support for {{status
"tasks"}} and its backing support in ClusterStatus, exposing a recent snapshot
of per server MonitoredTasks to the admin user. It's not a polished summary
presentation but that seems not required. Reopen or file a follow up if desired.
> Improve region housekeeping status observability
>
>
> Key: HBASE-24527
> URL: https://issues.apache.org/jira/browse/HBASE-24527
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Compaction, Operability, shell, UI
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> We provide a coarse grained admin API and associated shell command for
> determining the compaction status of a table:
> {noformat}
> hbase(main):001:0> help "compaction_state"
> Here is some help for this command:
> Gets compaction status (MAJOR, MAJOR_AND_MINOR, MINOR, NONE) for a table:
> hbase> compaction_state 'ns1:t1'
> hbase> compaction_state 't1'
> {noformat}
> We also log compaction activity, including a compaction journal at
> completion, via log4j to whatever log aggregation solution is available in
> production.
> This is not sufficient for online and interactive observation, debugging, or
> performance analysis of current compaction activity. In this kind of activity
> an operator is attempting to observe and analyze compaction activity in real
> time. Log aggregation and presentation solutions have typical latencies (end
> to end visibility of log lines on the order of ~minutes) which make that not
> possible today.
> We don't offer any API or tools for directly interrogating split and merge
> activity in real time. Some indirect knowledge of split or merge activity can
> be inferred from RIT information via ClusterStatus. It can also be scraped,
> with some difficulty, from the debug servlet.
> We should have new APIs and shell commands, and perhaps also new admin UI
> views, for
> at regionserver scope:
> * listing the current state of a regionserver's compaction, split, and merge
> tasks and threads
> * counting (simple view) and listing (detailed view) a regionserver's
> compaction queues
> * listing a region's currently compacting, splitting, or merging status
> at master scope, aggregations of the above detailed information into:
> * listing the active compaction tasks and threads for a given table, the
> extension of _compaction_state_ with a new detailed view
> * listing the active split or merge tasks and threads for a given table's
> regions
> Compaction detail should include the names of the effective engine and policy
> classes, and the results and timestamp of the last compaction selection
> evaluation. Split and merge detail should include the names of the effective
> policy classes and the result of the last split or merge criteria evaluation.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-22253) An AuthenticationTokenSecretManager leader won't step down if another RS claims to be a leader
[
https://issues.apache.org/jira/browse/HBASE-22253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-22253.
-
Assignee: (was: Esteban Gutierrez)
Resolution: Incomplete
> An AuthenticationTokenSecretManager leader won't step down if another RS
> claims to be a leader
> --
>
> Key: HBASE-22253
> URL: https://issues.apache.org/jira/browse/HBASE-22253
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0-alpha-1, 2.1.0, 2.2.0
>Reporter: Esteban Gutierrez
>Priority: Critical
>
> We ran into a situation were a rogue Lily HBase Indexer [SEP
> Consumer|https://github.com/NGDATA/hbase-indexer/blob/master/hbase-sep/hbase-sep-impl/src/main/java/com/ngdata/sep/impl/SepConsumer.java#L169]
> sharing the same {{zookeeper.znode.parent}} claimed to be
> AuthenticationTokenSecretManager for an HBase cluster. This situation
> undesirable since the leader running on the HBase cluster doesn't steps down
> when the rogue leader registers in the HBase cluster and both will start
> rolling keys with the same IDs causing authentication errors. Even a
> reasonable "fix" is to point to a different {{zookeeper.znode.parent}}, we
> should make sure that we step down as leader correctly.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-22229) Update SLB to take into account region server capacities
[ https://issues.apache.org/jira/browse/HBASE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-9. - Resolution: Incomplete > Update SLB to take into account region server capacities > > > Key: HBASE-9 > URL: https://issues.apache.org/jira/browse/HBASE-9 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Biju Nair >Priority: Major > Attachments: Balancing-heterogenous-cluster.pdf > > > Currently HBase SLB doesn't know about the physical capacity of region > servers and assumes that all the servers in the cluster are of same capacity. > But in reality cluster will have servers with different configuration. It > will be a good improvement if SLB can take into account the variations in > the capacity of different servers. Note that there is another ticket > HBASE-11780 similar to this to handle heterogenous clusters but to add a > resource based load balancer. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-20933) multiple splits may result into forever uncleaned split region
[ https://issues.apache.org/jira/browse/HBASE-20933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-20933. - Resolution: Fixed Addressed by subtasks > multiple splits may result into forever uncleaned split region > -- > > Key: HBASE-20933 > URL: https://issues.apache.org/jira/browse/HBASE-20933 > Project: HBase > Issue Type: Bug >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Major > Attachments: HBASE-20933.patch, Test123.java > > > Incase of multiple subsequent split and with an open handle on old reference > file, it may result into split region which can never be cleaned > So Here are two issues. > # Region is getting split even when it has reference to its parent > # Region is going offline/in archive mode even when there are reference > pending in store > *Repro Steps* > # Region split (P) > # Before major compaction starts after split, open a handle on store file on > new region (DA & DB) > # Let compaction completes on DA, (Here compaction will not clear reference > store files as it is opened) > # Split new region (DA) again ( shouldSplit will return true as before > compaction even does the cleanup, it removes the compacted files and > reference in-memory list) > # Now CatalogJanitor will not remove this region as it has store references, > majorCompaction/CompactedHFilesDischarger will not do the cleanup as it only > looks at only online regions > # After above steps region-DA which is offline will always be in split > regions and never getting cleaned up. > We found that catalog janitor is also not able to clean regions which are > offline(split parent) because it has reference of the daughter of it's parent > which is not getting cleaned up. This is causing lot of store files not > getting cleaned causing more space in local index store and lot of split > lingering regions. > Unit test repro the scenario has been attached. > Fix can be in CompactedHFilesDischarger or catalogJanitor to handle such > cases. Even if such region exists which are offline and are split region. > They should be able to clean t hem selves -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27115) [JDK11] Support JDK11 LTS in HBase Connectors
Nick Dimiduk created HBASE-27115: Summary: [JDK11] Support JDK11 LTS in HBase Connectors Key: HBASE-27115 URL: https://issues.apache.org/jira/browse/HBASE-27115 Project: HBase Issue Type: Task Components: build Reporter: Nick Dimiduk It seems that we also have some work to do in the hbase-connectors repo. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-26856) BufferedDataBlockEncoder.OnheapDecodedCell value can get corrupted
[
https://issues.apache.org/jira/browse/HBASE-26856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk resolved HBASE-26856.
--
Resolution: Fixed
> BufferedDataBlockEncoder.OnheapDecodedCell value can get corrupted
> --
>
> Key: HBASE-26856
> URL: https://issues.apache.org/jira/browse/HBASE-26856
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13
>
>
> In our production cluster we observed the cell value is modified after
> successful scanner read. After analyzing we observed OnheapDecodedCell is not
> created properly.
> We create OnheapDecodedCell with complete valAndTagsBuffer underlying array.
> {code:java}
> return new OnheapDecodedCell(Bytes.copy(keyBuffer, 0, this.keyLength),
> currentKey.getRowLength(), currentKey.getFamilyOffset(),
> currentKey.getFamilyLength(),
> currentKey.getQualifierOffset(), currentKey.getQualifierLength(),
> currentKey.getTimestamp(), currentKey.getTypeByte(),
> valAndTagsBuffer.array(),
> valAndTagsBuffer.arrayOffset() + vOffset, this.valueLength,
> memstoreTS, tagsArray,
> tOffset, this.tagsLength);
> {code}
> Here we are passing valAndTagsBuffer.array() for value extraction.
> The underlying array will be modified if it is altered anywhere.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (HBASE-27114) Upgrade scalatest maven plugin for thread-safety
Nick Dimiduk created HBASE-27114:
Summary: Upgrade scalatest maven plugin for thread-safety
Key: HBASE-27114
URL: https://issues.apache.org/jira/browse/HBASE-27114
Project: HBase
Issue Type: Task
Components: build, spark
Affects Versions: hbase-connectors-1.0.1, hbase-connectors-1.1.0
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
The {{master}} branch on the connectors repo warns when {{--threads}} is
issued, the complaint being the scalatest-maven-plugin. Looks like the latest
version resolves the complaint. Let's upgrade.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-26949) Purge references to ports 60010 and 60030 from the book
[ https://issues.apache.org/jira/browse/HBASE-26949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-26949. -- Resolution: Fixed Thanks for the cleanup [~itstitus1] ! > Purge references to ports 60010 and 60030 from the book > --- > > Key: HBASE-26949 > URL: https://issues.apache.org/jira/browse/HBASE-26949 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 3.0.0-alpha-3 >Reporter: Nick Dimiduk >Assignee: Titus Thomas >Priority: Minor > Fix For: 3.0.0-alpha-3 > > > Pre-1.0 versions of HBase have been gone for a good long time. Take a pass > over the book and update language around specific port numbers and simplify. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-14583) Enabled client-side metrics by default
[ https://issues.apache.org/jira/browse/HBASE-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-14583. -- Resolution: Won't Fix Zombies, huh? Let me try to repro that behavior. As for making this the default, I think better not. My current thinking is that we'll start using up client-side resources, which may be more precious than than server-side. Anyone watching here really wishing this was enabled by default? Please re-open. > Enabled client-side metrics by default > -- > > Key: HBASE-14583 > URL: https://issues.apache.org/jira/browse/HBASE-14583 > Project: HBase > Issue Type: Task > Components: Client, Operability, Performance >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Critical > Attachments: 14583.00.branch-1.patch, 14583.00.patch > > > Enabling this feature by default for master and branch-1. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-27106) HBase site generation should use its own JRuby
[
https://issues.apache.org/jira/browse/HBASE-27106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk resolved HBASE-27106.
--
Resolution: Fixed
I don't see the asciidoctor plugin on branch-2, so I assume we don't need to
backport this change anywhere.
> HBase site generation should use its own JRuby
> --
>
> Key: HBASE-27106
> URL: https://issues.apache.org/jira/browse/HBASE-27106
> Project: HBase
> Issue Type: Bug
> Components: website
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0-alpha-3
>
>
> The maven plugin we use to generate the reference guide,
> {{asciidoctor-maven-plugin}}, does not yet work with JRuby 9.3.
> Due to a dependency error we currently get a mix of JRuby 9.2 and 9.3
> libraries when building the site goal, which fails without an actionable
> error message. Correcting the dependency to consistently get JRuby 9.3 also
> results in failure, but due to gem resolution.
> We originally started overriding the JRuby version to get Aarch64 support.
> The version that ships with the plugin has this support already.
> Falling back to JRuby 9.2 for the plugin will mean our reference guide
> building won't work on M1 systems, but I don't see how we can change that
> until the plugin updates for JRuby 9.3 changes.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Reopened] (HBASE-27106) HBase site generation should use its own JRuby
[
https://issues.apache.org/jira/browse/HBASE-27106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk reopened HBASE-27106:
--
> HBase site generation should use its own JRuby
> --
>
> Key: HBASE-27106
> URL: https://issues.apache.org/jira/browse/HBASE-27106
> Project: HBase
> Issue Type: Bug
> Components: website
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0-alpha-3
>
>
> The maven plugin we use to generate the reference guide,
> {{asciidoctor-maven-plugin}}, does not yet work with JRuby 9.3.
> Due to a dependency error we currently get a mix of JRuby 9.2 and 9.3
> libraries when building the site goal, which fails without an actionable
> error message. Correcting the dependency to consistently get JRuby 9.3 also
> results in failure, but due to gem resolution.
> We originally started overriding the JRuby version to get Aarch64 support.
> The version that ships with the plugin has this support already.
> Falling back to JRuby 9.2 for the plugin will mean our reference guide
> building won't work on M1 systems, but I don't see how we can change that
> until the plugin updates for JRuby 9.3 changes.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
