[jira] [Resolved] (HBASE-24768) Clear cached service kerberos ticket in case of SASL failures thrown from server side
[ https://issues.apache.org/jira/browse/HBASE-24768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby resolved HBASE-24768. - Fix Version/s: 1.7.0 Resolution: Fixed This JIRA was merged back in October 2020 and seems to have been included in 1.7.0, but wasn't resolved and didn't have a Fix Version. > Clear cached service kerberos ticket in case of SASL failures thrown from > server side > - > > Key: HBASE-24768 > URL: https://issues.apache.org/jira/browse/HBASE-24768 > Project: HBase > Issue Type: Bug >Reporter: Sandeep Guggilam >Priority: Major > Fix For: 1.7.0 > > > We setup a SASL connection using different mechanisms like Digest, Kerberos > from master to RS for various activities like region assignment etc. In case > of SASL connect failures, we try to dispose of the SaslRpcClient and try to > relogin from the keytab on the client side. However the relogin from keytab > method doesn't clear off the service ticket cached in memory unless TGT is > about to expire within a timeframe. > This actually causes an issue where there is a keytab refresh that happens > because of expiry on the RS server and throws a SASL connect error when > Master reaches out to the RS server with the cached service ticket that no > longer works with the new refreshed keytab. We might need to clear off the > service ticket cached as there could be a credential refresh on the RS server > side when handling connect failures -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-26115) ServerTestHBaseCluster interface for testing coprocs
Geoffrey Jacoby created HBASE-26115: --- Summary: ServerTestHBaseCluster interface for testing coprocs Key: HBASE-26115 URL: https://issues.apache.org/jira/browse/HBASE-26115 Project: HBase Issue Type: Test Affects Versions: 3.0.0-alpha-1 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby The new TestingHBaseCluster introduced in HBASE-26080 provides a clean way for downstream developers writing features using the HBase client APIs to test their code. Its inner minicluster class, HBaseTestingUtil, was left unexposed with an interface audience of Phoenix, because coprocessors might need access to the internals of HBase itself to be tested. Occasionally, a developer outside of HBase and Phoenix might need the same access to the internals. One way to do this would be to introduce a new interface, ServerTestHBaseCluster, that extends TestingHBaseCluster and exposes the HBaseTestingUtil, with an interface audience of COPROC, REPLICATION (for custom endpoints), PHOENIX. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26102) TestingHBaseCluster should support running MapReduce jobs
Geoffrey Jacoby created HBASE-26102: --- Summary: TestingHBaseCluster should support running MapReduce jobs Key: HBASE-26102 URL: https://issues.apache.org/jira/browse/HBASE-26102 Project: HBase Issue Type: Test Components: test Affects Versions: 3.0.0-alpha-1 Reporter: Geoffrey Jacoby The existing HBaseTestingUtility optionally allows users to start a mini-YARN cluster for the purpose of running MapReduce jobs, such as testing applications that use HBase's MapReduce integration. The new TestingHBaseCluster is a cleaner interface than HBTU, but exposes less functionality, and in particular does not seem to allow for starting a YARN minicluster. This would make it difficult to use in some common circumstances, such as testing Phoenix's index builds, which use MapReduce. Many downstream projects that depend on HBase may also have batch logic done as MapReduce or other kinds of YARN jobs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25751) Add writable TimeToPurgeDeletes to ScanOptions
Geoffrey Jacoby created HBASE-25751: --- Summary: Add writable TimeToPurgeDeletes to ScanOptions Key: HBASE-25751 URL: https://issues.apache.org/jira/browse/HBASE-25751 Project: HBase Issue Type: New Feature Reporter: Geoffrey Jacoby In HBase 1.x and before, it was possible to override all settings on the ScanInfo when overriding the flush and compaction scanner open coprocoessors. HBase 2.x changed the coprocessor model to instead allow changes to a limited set of options on the ScanOptions object. In HBASE-19895 and HBASE-24321, we added support for KeepDeletedCells and MinVersions to be overriden by ScanOptions, as needed by Phoenix. A 1.x coprocessor used at my day job overrides TimeToPurgeDeletes, and to convert it to HBase 2.x that property would need to be overridable from ScanOptions as well. This should be a straightforward extension of the previous work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25740) Backport HBASE-25629 to branch-1
Geoffrey Jacoby created HBASE-25740: --- Summary: Backport HBASE-25629 to branch-1 Key: HBASE-25740 URL: https://issues.apache.org/jira/browse/HBASE-25740 Project: HBase Issue Type: Test Affects Versions: 1.7.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby HBASE-25629 recently fixed an issue where TestCurrentHourProvider consistently failed on certain OSes due to quirks in time zone implementations. This test is also failing in branch-1, so in order to expedite a potential 1.7.0 release we should backport to branch-1 as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25731) TestConnectionImplementation BadHostname tests fail in branch-1
Geoffrey Jacoby created HBASE-25731: --- Summary: TestConnectionImplementation BadHostname tests fail in branch-1 Key: HBASE-25731 URL: https://issues.apache.org/jira/browse/HBASE-25731 Project: HBase Issue Type: Test Affects Versions: 1.7.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby TestConnectionImplementation.testGetAdminBadHostname and testGetClientBadHostname are consistently failing in branch-1. This is because they're assuming that the validity of the host is checked immediately upon getting the protobuf service object, when instead the service code purposefully waits until the first service call to check. I'll revise the tests to make service calls and verify that they return the correct exceptions (ServiceException wrapping an UnknownHostException). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25525) WALKey Extended Attributes don't serialize to ReplicationSink
Geoffrey Jacoby created HBASE-25525: --- Summary: WALKey Extended Attributes don't serialize to ReplicationSink Key: HBASE-25525 URL: https://issues.apache.org/jira/browse/HBASE-25525 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 1.6.0, 1.5.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby Fix For: 1.7.0 HBASE-22622 introduced extended attributes on the WALKey object and protobuf, and HBASE-22623 created a coprocessor hook, preWALAppend, so that coprocessors can create and insert their own extended attributes. These attributes are readable on the source-side, such as in a custom ReplicationEndpoint. However, in branch-1 ReplicationProtbufUtil doesn't correctly populate the extended attributes on the WALKey protobuf. (In 2.1+, HBASE-20625 incidentally fixes this as part of a larger refactoring of WALCellCodec logic.) This means that a custom ReplicationSink can't read extended attributes on a WALKey. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24909) Remove unnecessary ClassNotFoundException and NoSuchMethodException stack traces from HBase startup logs
Geoffrey Jacoby created HBASE-24909: --- Summary: Remove unnecessary ClassNotFoundException and NoSuchMethodException stack traces from HBase startup logs Key: HBASE-24909 URL: https://issues.apache.org/jira/browse/HBASE-24909 Project: HBase Issue Type: Improvement Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby During cluster startup there are a couple of places where HBase needs to use reflection to determine what version of Hadoop or protobuf it's using, and then instantiate the right class. If its first guess is wrong, it logs a ClassNotFoundException (for protobuf MessageLite) or NoSuchMethodException (for DFSClient.decryptEncryptedDataEncryptionKey). While this is done at DEBUG level, the exception types and the stack traces can lead to operators thinking something is wrong. Since neither the exception nor the stack trace is actually of interest, we should just log a brief, clear message at DEBUG level about which decision was made and eat the exception without logging it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24780) Index rebuilds should record scan timestamp to job counter
[ https://issues.apache.org/jira/browse/HBASE-24780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby resolved HBASE-24780. - Resolution: Invalid (sigh) Wrong project, apologies for the noise. > Index rebuilds should record scan timestamp to job counter > -- > > Key: HBASE-24780 > URL: https://issues.apache.org/jira/browse/HBASE-24780 > Project: HBase > Issue Type: Improvement >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > > The index tool output tables (PHOENIX_INDEX_TOOL and > PHOENIX_INDEX_TOOL_RESULT) are both keyed on the effective scan time of the > index rebuild. This makes the table quite difficult to query if you know the > index you're interested in but not the exact millisecond it was last rebuilt. > Short term: record this timestamp in the Phoenix index rebuild job counters > (this JIRA) > Longer term: we might want to consider reversing the PK to both cut down on > hotspotting and make the table easier to query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24780) Index rebuilds should record scan timestamp to job counter
Geoffrey Jacoby created HBASE-24780: --- Summary: Index rebuilds should record scan timestamp to job counter Key: HBASE-24780 URL: https://issues.apache.org/jira/browse/HBASE-24780 Project: HBase Issue Type: Improvement Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby The index tool output tables (PHOENIX_INDEX_TOOL and PHOENIX_INDEX_TOOL_RESULT) are both keyed on the effective scan time of the index rebuild. This makes the table quite difficult to query if you know the index you're interested in but not the exact millisecond it was last rebuilt. Short term: record this timestamp in the Phoenix index rebuild job counters (this JIRA) Longer term: we might want to consider reversing the PK to both cut down on hotspotting and make the table easier to query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24321) Add writable MinVersions and read-only Scan to coproc ScanOptions
Geoffrey Jacoby created HBASE-24321: --- Summary: Add writable MinVersions and read-only Scan to coproc ScanOptions Key: HBASE-24321 URL: https://issues.apache.org/jira/browse/HBASE-24321 Project: HBase Issue Type: Improvement Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby Fix For: 2.3.0 Between HBase 1.x and 2.0, the RegionObserver pre*ScannerOpen coprocessors were significantly changed so that the coproc implementer no longer has access to the actual Scanner, just a ScanOptions object that can be changed in limited ways. This is safer and prevents resource leaks and other bugs. While ScanOptions provides support for changing TTL, KeepDeletedCells, and MaxVersions, a fourth column family config parameter, MinVersions, appears to have been missed. This prevents coproc implementers from changing MinVersions dynamically. An example of this is PHOENIX-5645, which in the forthcoming Phoenix 4.16 (based on HBase 1.x) will allow users to configure a moving window where all versions are kept, and thus point-in-time queries are safe. This cannot be put in the forthcoming Phoenix 5.1 (based on HBase 2.1 and 2.2) because of the coproc changes. Relatedly, preStoreScannerOpen lacks access to the Scan in HBase 2.0 and up. This prevents coprocs from reading the Scan parameters to check if, for example, a Scan has set the max time to a point in the past, and thus needs to override KeepDeletedCells. This can lead to incorrect behavior when doing point-in-time queries or using transactional engines that treat physically committed HBase writes as logically uncommitted parts of a transaction. It's also a correctness problem for PHOENIX-5645. Please note that only _read-only_ access to the Scan from the store scanner coproc hook is in scope for this change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-24018) Access check for getTableDescriptors is too restrictive
[ https://issues.apache.org/jira/browse/HBASE-24018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby reopened HBASE-24018: - > Access check for getTableDescriptors is too restrictive > --- > > Key: HBASE-24018 > URL: https://issues.apache.org/jira/browse/HBASE-24018 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Priority: Major > > Currently getTableDescriptor requires a user to have Admin or Create > permissions. A client might need to get table descriptors to act accordingly > eg. based on an attribute set or a CP loaded. It should not be necessary for > the client to have create or admin privileges just to read the descriptor, > execute and/or read permission should be sufficient? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23766) Support Point-In-Time Queries
Geoffrey Jacoby created HBASE-23766: --- Summary: Support Point-In-Time Queries Key: HBASE-23766 URL: https://issues.apache.org/jira/browse/HBASE-23766 Project: HBase Issue Type: New Feature Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby HBase currently offers a snapshot feature which allows operators to capture the state of a table at a point in time in a way that can be cloned or queried in the future. It's quite useful in some circumstances, but limited because it's a heavyweight operation, and because it requires prior knowledge of the time you want to capture. Phoenix currently offers a feature called "SCN", which uses the max timestamp on Scans to provide the illusion of a "lookback" query at a point in time. It's imperfect, however, because of HBase's filtering and cleanup logic for deletes, max versions and TTLs can prevent users from seeing certain Cells they would have been able to see at a previous point in time. Even PHOENIX-5645, and the equivalent HBASE-23602, which try to control major compaction cleanup, don't cover all edge cases completely. (For example, you can't see rows whose TTL has expired now but hadn't back then. Same with max versions.) There are useful non-Phoenix applications as well, such as a change stream that shows before/after images, as DynamoDB offers. Since full support will require new configuration options added not just to major compaction, but also to the read pipeline, I'm filing this as an umbrella JIRA so we can have smaller sub-tasks, rather than trying to cram everything into HBASE-23602. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23744) FastPathBalancedQueueRpcExecutor should enforce queue length of 0
Geoffrey Jacoby created HBASE-23744: --- Summary: FastPathBalancedQueueRpcExecutor should enforce queue length of 0 Key: HBASE-23744 URL: https://issues.apache.org/jira/browse/HBASE-23744 Project: HBase Issue Type: Bug Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby FastPathBalancedQueueRpcExecutor allows RPC requests to skip the RPC queue and get worked by an available handler under certain circumstances. Relatedly, the hbase.ipc.server.max.callqueue.length parameter can be set to 0, including dynamically. When this is the case the executor is supposed to block all dispatching. However, the FastPathBalancedQueueRpcExecutor will still dispatch the request if one of the "fast path" handlers is available on its stack. This both isn't the desired effect, and also makes TestSimpleRpcExecutor.testSoftAndHardQueueLimits unstable when it checks the queue length 0 behavior. A simple fix is just to check max queue length > 0 before FastPathBalancedQueueRpcExecutor pops the fast handler off the stack. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-22908) Link to HBase 1.x Documentation on HBase Site
[ https://issues.apache.org/jira/browse/HBASE-22908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby resolved HBASE-22908. - Resolution: Fixed I don't know which kind soul added the 1.4 documentation to the HBase website, but it's there now, so resolving this JIRA. > Link to HBase 1.x Documentation on HBase Site > - > > Key: HBASE-22908 > URL: https://issues.apache.org/jira/browse/HBASE-22908 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Geoffrey Jacoby >Priority: Minor > > Some other communities in the Hadoop stack, such as Hadoop itself and > ZooKeeper, maintain links to documentation for all of their active branches. > (ZooKeeper additionally has an archive to documentation on all their previous > releases) > Since I believe the stable pointer is still on HBase 1.x, it's odd that the > HBase site does not actually provide a link to documentation for it, only > newer releases 2.0 and up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23711) Add test for MinVersions and KeepDeletedCells TTL
Geoffrey Jacoby created HBASE-23711: --- Summary: Add test for MinVersions and KeepDeletedCells TTL Key: HBASE-23711 URL: https://issues.apache.org/jira/browse/HBASE-23711 Project: HBase Issue Type: Test Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby Recently I was researching how HBase handles the interactions between setting MinVersions and KeepDeletedCells = TTL, and I wrote a test to prove my assumptions about the behavior were correct. There doesn't seem to be an equivalent existing test in TestMinVersions, so I thought I'd contribute it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23710) Priority configuration for system coprocessors
Geoffrey Jacoby created HBASE-23710: --- Summary: Priority configuration for system coprocessors Key: HBASE-23710 URL: https://issues.apache.org/jira/browse/HBASE-23710 Project: HBase Issue Type: New Feature Components: Coprocessors Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby Currenty HBase allows operators to set system region coprocessors via hbase-site.xml to be loaded on each table in a cluster (or alternately, all tables but system tables). HBase assumes that the first loaded system coprocessor gets the first, or SYSTEM priority, with each subsequent system coproc getting incremented by 1. As a reminder, in HBase _lower_ priorities go first. It can be useful for an operator to be able to define a coprocessor on each table that needs a different priority. For example, an operator might want a coproc to load on each table _last_, so that it can enforce some system invariant and know that no other coproc will interfere with it. I propose adding optional priority config to the hbase-site.xml configuration, separated from each coproc class in the comma-separated list by a special character (perhaps a colon) that's not used in class names. The region coprocessor host will parse the priority if present and use it when instantiating the coproc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23602) TTL Before Which No Data is Purged
Geoffrey Jacoby created HBASE-23602: --- Summary: TTL Before Which No Data is Purged Key: HBASE-23602 URL: https://issues.apache.org/jira/browse/HBASE-23602 Project: HBase Issue Type: New Feature Reporter: Geoffrey Jacoby HBase currently offers operators a choice. They can set KEEP_DELETED_CELLS=true and VERSIONS to max value, plus no TTL, and they will always have a complete history of all changes (but high storage costs and penalties to read performance). Or they can have KEEP_DELETED_CELLS=false and VERSIONS/TTL set to some reasonable values, but that means that major compactions can destroy the ability to do a consistent snapshot read of any prior time. (This limits the usefulness and correctness of, for example, Phoenix's SCN lookback feature.) I propose having a new TTL property to give a minimum age that an expired or deleted Cell would have to achieve before it could be purged. (I see that HBASE-10118 already does something similar for the delete markers themselves.) This would allow operators to have a consistent history for some finite amount of recent time while still purging out the "long tail" of obsolete / deleted versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23316) RegionServers should refuse to load Regions with malformed coprocs, but not crash
Geoffrey Jacoby created HBASE-23316: --- Summary: RegionServers should refuse to load Regions with malformed coprocs, but not crash Key: HBASE-23316 URL: https://issues.apache.org/jira/browse/HBASE-23316 Project: HBase Issue Type: Improvement Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby Currently, a region server will crash if it tries to load a region with a coprocessor that is malformed (such as not being on the RS's classpath.) This can lead to a cascading "poison pill" in which the HMaster keeps reassigning the region to different region servers, bringing down server after server and endangering the whole cluster. We definitely can't load the Region if the coproc is wrong, but neither should that harm other, correctly configured regions on the same server. In this JIRA, I'll change the behavior to fail to load the region, and increment a metric for region load failures. Future JIRAs can build on this, such as by having the HMaster stop trying to load a malformed region absent user intervention after some number of retries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23288) Backport HBASE-23251 (Add Column Family and Table Names to HFileContext) to branch-1
Geoffrey Jacoby created HBASE-23288: --- Summary: Backport HBASE-23251 (Add Column Family and Table Names to HFileContext) to branch-1 Key: HBASE-23288 URL: https://issues.apache.org/jira/browse/HBASE-23288 Project: HBase Issue Type: Sub-task Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23251) Add Region and Table Names to HFileContext and use in HFileWriterImpl logging
Geoffrey Jacoby created HBASE-23251: --- Summary: Add Region and Table Names to HFileContext and use in HFileWriterImpl logging Key: HBASE-23251 URL: https://issues.apache.org/jira/browse/HBASE-23251 Project: HBase Issue Type: Improvement Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby When something goes wrong in the Store / HFile write path, it would be very useful to know which region and table the error is coming from. Currently the HFileWriterImpl gets an HFileContext object with some useful state information, but the region and table aren't among them. For example, this would be very helpful diagnosing HBASE-23143 and similar issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-22934) IndexedKeyValue creation fails after HBASE-22034
[ https://issues.apache.org/jira/browse/HBASE-22934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby resolved HBASE-22934. - Resolution: Invalid > IndexedKeyValue creation fails after HBASE-22034 > > > Key: HBASE-22934 > URL: https://issues.apache.org/jira/browse/HBASE-22934 > Project: HBase > Issue Type: Improvement >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > > HBASE-22034 backported to branch-1 HBASE-21401 (adding several validation > checks to KeyValue creation) and HBASE-22032 (adding a null check for the row > key to KeyValue creation). It will first be released in HBase 1.5. > In Phoenix 4.14.2 we included PHOENIX-5188, which adds logic to initialize > IndexedKeyValues with the appropriate row key and offsets, so that it can > pass the new check in HBASE-22032. However, it did not correctly handle all > the checks in HBASE-21401. > When testing 4.14.3 of Phoenix with an HBase version containing HBASE-22034, > we see lots of errors like the following: > java.lang.IllegalArgumentException: Overflow when reading key length at > position=0, KeyValueBytesHex=foo, offset=0, length=3 > This will affect the future 4.15-HBase-1.5, and as well as any future release > of a 4.14.3 based on HBase-1.5 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (HBASE-22934) IndexedKeyValue creation fails after HBASE-22034
Geoffrey Jacoby created HBASE-22934: --- Summary: IndexedKeyValue creation fails after HBASE-22034 Key: HBASE-22934 URL: https://issues.apache.org/jira/browse/HBASE-22934 Project: HBase Issue Type: Improvement Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby HBASE-22034 backported to branch-1 HBASE-21401 (adding several validation checks to KeyValue creation) and HBASE-22032 (adding a null check for the row key to KeyValue creation). It will first be released in HBase 1.5. In Phoenix 4.14.2 we included PHOENIX-5188, which adds logic to initialize IndexedKeyValues with the appropriate row key and offsets, so that it can pass the new check in HBASE-22032. However, it did not correctly handle all the checks in HBASE-21401. When testing 4.14.3 of Phoenix with an HBase version containing HBASE-22034, we see lots of errors like the following: java.lang.IllegalArgumentException: Overflow when reading key length at position=0, KeyValueBytesHex=foo, offset=0, length=3 This will affect the future 4.15-HBase-1.5, and as well as any future release of a 4.14.3 based on HBase-1.5 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (HBASE-22908) Link to HBase 1.x HBase Documentation on HBase Site
Geoffrey Jacoby created HBASE-22908: --- Summary: Link to HBase 1.x HBase Documentation on HBase Site Key: HBASE-22908 URL: https://issues.apache.org/jira/browse/HBASE-22908 Project: HBase Issue Type: Improvement Components: documentation Reporter: Geoffrey Jacoby Some other communities in the Hadoop stack, such as Hadoop itself and ZooKeeper, maintain links to documentation for all of their active branches. (ZooKeeper additionally has an archive to documentation on all their previous releases) Since I believe the stable pointer is still on HBase 1.x, it's odd that the HBase site does not actually provide a link to documentation for it, only newer releases 2.0 and up. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (HBASE-22623) Add coprocessor hooks for preWALAppend and postWALAppend
Geoffrey Jacoby created HBASE-22623: --- Summary: Add coprocessor hooks for preWALAppend and postWALAppend Key: HBASE-22623 URL: https://issues.apache.org/jira/browse/HBASE-22623 Project: HBase Issue Type: New Feature Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby While many coprocessor hooks expose the WALEdit to implementing coprocs, there aren't any that expose the WALKey before it's created and added to the WALEntry. It's sometimes useful for coprocessors to be able to edit the WALKey, for example to add extended attributes using the fields to be added in HBASE-22622. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22622) WALKey Extended Attributes
Geoffrey Jacoby created HBASE-22622: --- Summary: WALKey Extended Attributes Key: HBASE-22622 URL: https://issues.apache.org/jira/browse/HBASE-22622 Project: HBase Issue Type: New Feature Components: wal Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby It would be useful if the WAL protobuf and WALKey class included an optional map of extended key/value attributes. While standard HBase replication would not use them, custom replication endpoints could use the data to make filtering decisions or take actions. An example use case would be allowing a tool like Phoenix to annotate WAL.Entries to indicate that a given Entry is associated with a particular Phoenix view rather than the base table. A custom replication endpoint might choose to replicate some views but not others. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22034) Backport HBASE-21404 and HBASE-22032 to branch-1
Geoffrey Jacoby created HBASE-22034: --- Summary: Backport HBASE-21404 and HBASE-22032 to branch-1 Key: HBASE-22034 URL: https://issues.apache.org/jira/browse/HBASE-22034 Project: HBase Issue Type: Improvement Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby Branch-2 and master have good validation checks when constructing KeyValues. We should also have them on branch-1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22032) KeyValue validation should check for null byte array
Geoffrey Jacoby created HBASE-22032: --- Summary: KeyValue validation should check for null byte array Key: HBASE-22032 URL: https://issues.apache.org/jira/browse/HBASE-22032 Project: HBase Issue Type: Improvement Affects Versions: 2.1.3, 2.0.4, 3.0.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby HBASE-21401 added some nice validation checks to throw precise errors if a KeyValue is constructed using invalid parameters. However it implicitly assumes that the KeyValue buffer is not null. It should validate this assumption and alert accordingly rather than throwing an NPE from an unrelated check. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21530) Abort_Procedure should be able to take a list of proc IDs
Geoffrey Jacoby created HBASE-21530: --- Summary: Abort_Procedure should be able to take a list of proc IDs Key: HBASE-21530 URL: https://issues.apache.org/jira/browse/HBASE-21530 Project: HBase Issue Type: Improvement Reporter: Geoffrey Jacoby As a convenience, it would be helpful if the HBase shell's abort_procedure call had the option of taking in multiple procedure ids at the same time, rather than relying on operators to use a loop in an external script. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21529) Abort_Procedure in HBase shell fails when all handlers are taken up by Procedures
Geoffrey Jacoby created HBASE-21529: --- Summary: Abort_Procedure in HBase shell fails when all handlers are taken up by Procedures Key: HBASE-21529 URL: https://issues.apache.org/jira/browse/HBASE-21529 Project: HBase Issue Type: Bug Components: proc-v2 Affects Versions: 1.4.8, 1.3.2 Reporter: Geoffrey Jacoby If all RPC handlers are taken up by stuck procedures, the operator will not be able to abort them, because the abort procedure call will try to use the same (exhausted) pool of RPC handlers. The abort procedure call should use a dedicated pool of RPC handlers. This pool can probably be small, but it needs to be separate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-17912) Avoid major compactions on region server startup
Geoffrey Jacoby created HBASE-17912: --- Summary: Avoid major compactions on region server startup Key: HBASE-17912 URL: https://issues.apache.org/jira/browse/HBASE-17912 Project: HBase Issue Type: Improvement Components: Compaction Affects Versions: 0.98.24, 1.3.1, 2.0.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby The HRegionServer.CompactionChecker chore wakes up every 10s and for each store in each region mods against a chore frequency (by default slightly under 3 hours) to see if it's time to check if a major compaction is necessary for that store. The check to see if it needs to check for major compactions is calculated by if (iteration % multiplier != 0) continue; where iteration is the number of times the chore has woken up. Because 0 % anything is 0, this will always check for necessary major compactions on each store when this chore is first run after the region server starts up. This can result in compaction storms when doing a rolling restart, because, for example, the new instance of the region server might get a lower jitter value than the old one had. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17543) Create additional ReplicationEndpoint WALEntryFilters by configuration
Geoffrey Jacoby created HBASE-17543: --- Summary: Create additional ReplicationEndpoint WALEntryFilters by configuration Key: HBASE-17543 URL: https://issues.apache.org/jira/browse/HBASE-17543 Project: HBase Issue Type: Improvement Components: Replication Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby The existing BaseReplicationEndpoint creates a ChainWALEntryFilter containing a NamespaceTableCfWALEntryFilter and a ScopeWALEntryFilter. Adding a custom WALEntryFilter type to this chain requires creating an entirely new ReplicationEndpoint subclass and creating a new peer on the running cluster, which can be operationally complex to transition to without data loss in cases such as master/master. For WALEntryFilters without constructor parameters, it would be straightforward to have a Configuration option to list additional WALEntryFilter classes the operator wants to include in the filter chain in the default endpoint, and then have the endpoint instantiate the filters via reflection. Then filter logic could be added (or removed) with only a hbase-site.xml change and a rolling restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16576) Shell add_peer doesn't allow setting cluster_key for custom endpoints
Geoffrey Jacoby created HBASE-16576: --- Summary: Shell add_peer doesn't allow setting cluster_key for custom endpoints Key: HBASE-16576 URL: https://issues.apache.org/jira/browse/HBASE-16576 Project: HBase Issue Type: Bug Components: shell Affects Versions: 0.98.22, 1.1.5, 2.0.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby The HBase shell allows a user to create a replication peer using the add_peer method, which can take a peer id and a Ruby hash. It creates a ReplicationPeerConfig and passes it through to the Java ReplicationAdmin#addPeer. The Ruby code makes an assumption that the Java API doesn't: that CLUSTER_KEY and ENDPOINT_CLASSNAME are mutually exclusive. If both are specified, it throws an error. If only ENDPOINT_CLASSNAME is set, the add_peer logic derives a local dummy cluster key based on the local cluster's configuration. CLUSTER_KEY shouldn't be required when an ENDPOINT_CLASSNAME is specified, because a custom endpoint might not need it. The dummy default logic is fine. But if an endpoint does require a remote cluster key, it shouldn't be forbidden to provide one, especially since the Java API permits it, and even the custom replication endpoint Java tests rely on this. (See TestReplicationEndpoint#testCustomReplicationEndpoint) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16448) Custom metrics for custom replication endpoints
Geoffrey Jacoby created HBASE-16448: --- Summary: Custom metrics for custom replication endpoints Key: HBASE-16448 URL: https://issues.apache.org/jira/browse/HBASE-16448 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 0.98.21, 1.2.2, 2.0.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby ReplicationEndpoints contain a MetricsSource class that can update a fixed set of key replication metrics. It does not, however, allow a developer implementing a custom ReplicationEndpoint to create/update custom metrics unique to the new endpoint's use case. As it turns out, MetricsSource wraps a class, which in turn wraps a class, that implements the Hadoop BaseSource interface, which does allow for custom metrics. Simply having the two wrappers implement BaseSource as well with pass-through methods should be straightforward, and would allow replication endpoints to use these generic metric methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16108) RowCounter should support multiple key ranges
Geoffrey Jacoby created HBASE-16108: --- Summary: RowCounter should support multiple key ranges Key: HBASE-16108 URL: https://issues.apache.org/jira/browse/HBASE-16108 Project: HBase Issue Type: Improvement Reporter: Geoffrey Jacoby Currently, RowCounter only allows a single key range to be used as a filter. It would be useful in some cases to be able to specify multiple key ranges (or prefixes) in the same job. (For example, counting over a set of Phoenix tenant ids in an unsalted table) This could be done by enhancing the existing key range parameter to take multiple start/stop row pairs. Alternately, a new --row-prefixes option could be added, similar to what HBASE-15847 did for VerifyReplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15847) VerifyReplication prefix filtering
Geoffrey Jacoby created HBASE-15847: --- Summary: VerifyReplication prefix filtering Key: HBASE-15847 URL: https://issues.apache.org/jira/browse/HBASE-15847 Project: HBase Issue Type: New Feature Components: Replication Reporter: Geoffrey Jacoby VerifyReplication currently lets a user verify data within a time range has been replicated to a particular peer. It can be useful to verify only data that starts with particular prefixes. (An example would be an unsalted multi-tenant Phoenix table where you wish to only verify data for particular tenants.) Add a new option to the VerifyReplication job to allow for a list of prefixes to be given. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15647) Backport HBASE-15507 to 0.98
Geoffrey Jacoby created HBASE-15647: --- Summary: Backport HBASE-15507 to 0.98 Key: HBASE-15647 URL: https://issues.apache.org/jira/browse/HBASE-15647 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 0.98.18 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby HBASE-15507 introduced an update_peer_config method to allow online changes to replication config. It depended on HBASE-11393, which so far exists only in master and can only be in 2.0 and above because of some incompatible changes to replication peer config serialization. To get this in 0.98 will require two things: 1. Port the PeerConfigTracker in HBASE-11393 to 0.98 without taking the incompatible tableCF serialization changes. 2. Port HBASE-15507's logic on top of this. Most of this work was done in HBASE-15633 for branch-1, but 0.98 is just different enough to need some tweaks of its own. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15633) Backport HBASE-15507 to branch-1
Geoffrey Jacoby created HBASE-15633: --- Summary: Backport HBASE-15507 to branch-1 Key: HBASE-15633 URL: https://issues.apache.org/jira/browse/HBASE-15633 Project: HBase Issue Type: New Feature Components: Replication, shell Affects Versions: 1.3.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby HBase-15507 introduced an update_peer_config method to allow online changes to replication config. It depended on HBASE-11393, which so far exists only in master and can only be in 2.0 and above because of some incompatible changes to replication peer config serialization. To get this in branch-1 will require two things: 1. Port the PeerConfigTracker in HBASE-11393 to branch-1 without taking the incompatible tableCF serialization changes. 2. Port HBASE-15507's logic on top of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15472) replication_admin_test creates a table it doesn't use
[ https://issues.apache.org/jira/browse/HBASE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby resolved HBASE-15472. - Resolution: Duplicate > replication_admin_test creates a table it doesn't use > - > > Key: HBASE-15472 > URL: https://issues.apache.org/jira/browse/HBASE-15472 > Project: HBase > Issue Type: Bug > Components: Replication, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Minor > Labels: replication, shell > Attachments: HBASE-15472.patch > > > I noticed while adding tests to replication_admin_test.rb for HBASE-12940 > that it creates an HBase table "hbase_shell_tests_table" that is never used > in any of the suite's tests. Removing the table creation statements speeds up > the suite locally from 1min 10s to 2s. > Note that until HBASE-14562 is worked, this test suite doesn't run as part of > the automatic test runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15567) TestReplicationShell broken by recent replication changes
Geoffrey Jacoby created HBASE-15567: --- Summary: TestReplicationShell broken by recent replication changes Key: HBASE-15567 URL: https://issues.apache.org/jira/browse/HBASE-15567 Project: HBase Issue Type: Bug Components: Replication, shell Affects Versions: 2.0.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby Priority: Minor Recent changes to the Ruby shell's add_peer method in HBASE-11393 have broken TestReplicationShell, which went unnoticed because it's currently Ignored as flaky. This test is useful when developing extensions to the replication shell commands, and should be kept working (and hopefully re-enabled in the near future.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15507) Online modification of enabled ReplicationPeerConfig
Geoffrey Jacoby created HBASE-15507: --- Summary: Online modification of enabled ReplicationPeerConfig Key: HBASE-15507 URL: https://issues.apache.org/jira/browse/HBASE-15507 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 1.2.0, 2.0.0, 1.3.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby It's currently possible to update the table CFs for a replication peer while it's running, but not the peer configuration or data maps introduced as part of custom replication endpoints in HBASE-11367. This means that if you have a custom endpoint that depends on some configuration parameter, you have to remove the peer and recreate it in order to change the param. In some use cases that's not possible without risking data loss. HBASE-11393, which will consolidate tableCFs in the same znode as the rest of ReplicationPeerConfig, may help here, but with or without it it still seems like there needs to be further work to add update config/data API support to ReplicationAdmin and a callback mechanism to notify the endpoints of config parameter changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15472) replication_admin_test creates a table it doesn't use
Geoffrey Jacoby created HBASE-15472: --- Summary: replication_admin_test creates a table it doesn't use Key: HBASE-15472 URL: https://issues.apache.org/jira/browse/HBASE-15472 Project: HBase Issue Type: Bug Components: Replication, shell Affects Versions: 1.2.0, 2.0.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby Priority: Minor I noticed while adding tests to replication_admin_test.rb for HBASE-12940 that it creates an HBase table "hbase_shell_tests_table" that is never used in any of the suite's tests. Removing the table creation statements speeds up the suite locally from 1min 10s to 2s. Note that until HBASE-14562 is worked, this test suite doesn't run as part of the automatic test runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14985) TimeRange constructors should set allTime when appropriate
Geoffrey Jacoby created HBASE-14985: --- Summary: TimeRange constructors should set allTime when appropriate Key: HBASE-14985 URL: https://issues.apache.org/jira/browse/HBASE-14985 Project: HBase Issue Type: Improvement Affects Versions: 0.98.16.1, 1.1.2 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby Priority: Minor The default TimeRange constructor creates a range from 0 to Long.MAX_VALUE and sets an allTime flag to true. This flag allows some performance optimizations when comparing or using TimeRanges. This flag is not set, however, if you call "new TimeRange(0L)" or "new TimeRange(0L, Long.MAX_VALUE)", even though both of these create a logically equivalent TimeRange to "new TimeRange()". Since TimeRanges are immutable and detecting this condition is trivial, we should set the flag automatically in the explicit constructors when appropriate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14416) HBase Backup/Restore Phase 2: Create plugin hooks for Backup tools
Geoffrey Jacoby created HBASE-14416: --- Summary: HBase Backup/Restore Phase 2: Create plugin hooks for Backup tools Key: HBASE-14416 URL: https://issues.apache.org/jira/browse/HBASE-14416 Project: HBase Issue Type: Sub-task Reporter: Geoffrey Jacoby Different organizations may already have their own backup tools for HBase, or wish to develop them in the future for their own particular use cases. It would be useful if HBase supported hooks to integrate those tools so that they could be configured and run in a standard way. In particular, the administrative interface to start a backup, restore, or merge should be decoupled from any particular implementation as much as possible, so that any implementation with similar capabilities can be substituted via configuration without needing to fork or modify the code. Ideally, it will also be possible to decouple the various components so that implementations can be mixed and matched. (For example, one could use the standard backup's functionality to track what needs to be backed up, but use a custom plugin to change the logic or storage format of the backup operation itself.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13796) ZKUtil doesn't clean quorum setting properly
Geoffrey Jacoby created HBASE-13796: --- Summary: ZKUtil doesn't clean quorum setting properly Key: HBASE-13796 URL: https://issues.apache.org/jira/browse/HBASE-13796 Project: HBase Issue Type: Bug Affects Versions: 0.98.12, 1.1.0, 1.0.1 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby Priority: Minor ZKUtil.getZooKeeperClusterKey is obviously trying to pull out the ZooKeeper quorum setting from the config object and remove several special characters from it. Due to a misplaced parenthesis, however, it's instead running the replace operation on the config setting _name_, HConstants.ZOOKEEPER_QUORUM, and not the config setting itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
