[ANNOUNCE] Apache HBase 1.3.0 is now available for download
The HBase team is happy to announce the immediate availability of HBase 1.3. 0! Apache HBase is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To learn more about HBase , see https://hbase.apache.org/. HBase 1.3.0 is the third minor release in the HBase 1.x line and includes approximately 1700 resolved issues. Notable new features include: - Date-based tiered compactions (HBASE-15181, HBASE-15339) - Maven archetypes for HBase client applications (HBASE-14877) - Throughput controller for flushes (HBASE-14969) - Controlled delay (CoDel) based RPC scheduler (HBASE-15136) - Bulk loaded HFile replication (HBASE-13153) - More improvements to Procedure V2 - Improvements to Multi WAL (HBASE-14457) - Many improvements and optimizations in metrics subsystem - Reduced memory allocation in RPC layer (HBASE-15177) - Region location lookups optimizations in HBase client and numerous bugfixes and performance improvements. The full list of issues is available at https://s.apache.org/hbase-1.3.0-jira Download through an ASF mirror near you: http://www.apache.org/dyn/closer.lua/hbase/1.3.0 The relevant checksums files are available at: https://www.apache.org/dist/hbase/1.3.0/hbase-1.3.0-src.tar.gz.mds https://www.apache.org/dist/hbase/1.3.0/hbase-1.3.0-bin.tar.gz.mds Project members signature keys can be found at https://www.apache.org/dist/hbase/KEYS PGP signatures are available at: https://www.apache.org/dist/hbase/1.3.0/hbase-1.3.0-src.tar.gz.asc https://www.apache.org/dist/hbase/1.3.0/hbase-1.3.0-bin.tar.gz.asc For instructions on verifying ASF release downloads, please see https://www.apache.org/dyn/closer.cgi#verify Question, comments, and problems are always welcome at: dev@hbase.apache.org . Cheers, The HBase Dev Team
Re: [VOTE] The 1st HBase 1.3.0 release candidate (RC0) is available
Thanks Stack. I expect starting merges for 1.3.1 in next coming weeks, I think there's a dozen or so patches waiting for review and backport, and the more large-scale testing we can get the better. Thanks for testing that kerberized setup Jerry. Looking at the patch though, I imagine older HBase releases should also suffer from it, if they run on 2.5.1 Hadoop? Also as was noted in the jira, people were able to reproduce this on recent builds of JDK 1.7, so it's not 1.8 specific. We do strive to support older Hadoop releases and should be able to run on 2.5.1 as such, but this bug in Hadoop Common have been resolved year and a half ago and people running production kerberized clusters should have upgraded Hadoop to deal with such issues. I think it might worth separate thread to discuss how to deal with it, should we just say that we support 2.5.1 but recommend something later, or should we say that we recommend newer versions for security setups, or should we just make broad claim that we support 2.5.1 but in certain deployments users will be subject to bugs in underlying dependencies. What do you think? -Mikhail On Mon, Jan 16, 2017 at 9:11 PM, Stack wrote: > Late +1 > > Checked signature and hash on src tgz. > Built tgz from src w/ jdk7. > Loaded some data. Verified it made it. > Browsed UI, logs, and docs. > > All looks good. > > St.Ack > > P.S. Sorry, didn't get a chance to deploy and run RC test at larger scale. > Will do for 1.3.1. Have done a bunch of larger scale ITBLLs w/ Chaos on > recent tips of 1.3 and it seemed sound. > > > On Fri, Jan 6, 2017 at 12:42 PM, Mikhail Antonov > wrote: > > > Hello everyone, > > > > I'm pleased to announce the first release candidate for HBase 1.3.0 is > > available to download and testing. > > > > Artifacts are available here: > > > > https://dist.apache.org/repos/dist/dev/hbase/1.3.0RC0/ > > > > Maven artifacts are available in the staging repository: > > > > https://repository.apache.org/content/repositories/orgapachehbase-1162/ > > > > All artifacts are signed with my code signing key 35A4ABE2, which is also > > in the project KEYS file at > > > > http://www.apache.org/dist/hbase/KEYS > > > > these artifacts correspond to commit hash > > > > e359c76e8d9fd0d67396456f92bcbad9ecd7a710 tagged as 1.3.0RC0. > > > > HBase 1.3.0 is the third minor release in the HBase 1.x line and includes > > approximately 1700 resolved issues. > > > > Notable new features include: > > > > - Date-based tiered compactions (HBASE-15181, HBASE-15339) > > - Maven archetypes for HBase client applications (HBASE-14877) > > - Throughput controller for flushes (HBASE-14969) > > - Controlled delay (CoDel) based RPC scheduler (HBASE-15136) > > - Bulk loaded HFile replication (HBASE-13153) > > - More improvements to Procedure V2 > > - Improvements to Multi WAL (HBASE-14457) > > - Many improvements and optimizations in metrics subsystem > > - Reduced memory allocation in RPC layer (HBASE-15177) > > - Region location lookups optimizations in HBase client > > > > and numerous bugfixes and performance improvements. > > > > The full list of issues addressed is available at > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje > > ctId=12310753&version=12332794 > > > > and also in the CHANGES.txt file in the root directory of the source > > tarball. > > > > Please take a few minutes to verify the release and vote on > > releasing it: > > > > [ ] +1 Release this package as Apache HBase 1.3.0 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > > > > This Vote will run for one week and close Fri Jan 13, 2017 22:00 PST. > > > > Thank you! > > Mikhail > > > -- Thanks, Michael Antonov
Re: SocketTimeoutException on regionservers
Timeout after waiting ten seconds to read from HDFS is no fun. Tell us more about your HDFS setup. You collect system metrics on disks? Machines are healthy all around? How frequent does this occur? Thanks for the question, S On Thu, Jan 12, 2017 at 10:18 AM, Tulasi Paradarami < tulasi.krishn...@gmail.com> wrote: > Hi, > > I noticed that Regionservers are raising following exceptions > intermittently that is manifesting itself as request timeouts on the client > side. HDFS is in a healthy state and there are no corrupted blocks (from > "hdfs fsck" results). Datanodes were not out of service when this error > occurs and GC on datanodes is usually around 0.3sec. > > Also, when these exceptions occur, HDFS metric "Send Data Packet Blocked On > Network Average Time" tends to go up. > > Here are the configured values for some of the relevant parameters: > dfs.client.socket-timeout: 10s > dfs.datanode.socket.write.timeout: 10s > dfs.namenode.avoid.read.stale.datanode: true > dfs.namenode.avoid.write.stale.datanode: true > dfs.datanode.max.xcievers: 8192 > > Any pointers towards what could be causing these exceptions is appreciated. > Thanks. > > CDH 5.7.2 > HBase 1.2.0 > > ---> Regionserver logs > > 2017-01-11 19:19:04,940 WARN > [PriorityRpcServer.handler=3,queue=1,port=60020] hdfs.BlockReaderFactory: > I/O error constructing remote block reader. > java.net.SocketTimeoutException: 1 millis timeout while waiting for > channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=/datanode3:27094 > remote=/datanode2:50010] > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO( > SocketIOWithTimeout.java:164) > ... > > 2017-01-11 19:19:04,995 WARN > [PriorityRpcServer.handler=11,queue=1,port=60020] hdfs.DFSClient: > Connection failure: Failed to connect to /datanode2:50010 for file > /hbase/data/default//ec9ca > java.net.SocketTimeoutException: 1 millis timeout while waiting for > channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=/datanode3:27107 > remote=/datanode2:50010] > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO( > SocketIOWithTimeout.java:164) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) > at > org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver. > readChannelFully(PacketReceiver.java:258) > at > org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully( > PacketReceiver.java:209) > at > org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead( > PacketReceiver.java:171) > at > org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver. > receiveNextPacket(PacketReceiver.java:102) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket( > RemoteBlockReader2.java:207) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read( > RemoteBlockReader2.java:156) > at > org.apache.hadoop.hdfs.BlockReaderUtil.readAll(BlockReaderUtil.java:32) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readAll( > RemoteBlockReader2.java:386) > at > org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode( > DFSInputStream.java:1193) > at > org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange( > DFSInputStream.java:1112) > at > org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1473) > at > org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1432) > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:89) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra( > HFileBlock.java:752) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset( > HFileBlock.java:1448) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl. > readBlockDataInternal(HFileBlock.java:1648) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl. > readBlockData(HFileBlock.java:1532) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock( > HFileReaderV2.java:445) > at > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader. > loadDataBlockWithScanInfo(HFileBlockIndex.java:261) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo( > HFileReaderV2.java:642) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo( > HFileReaderV2.java:622) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter( > StoreFileScanner.java:314) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner. > reseek(StoreFileScanner.java:226) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner. > enforceSeek(StoreFileScanner.java:437) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap. > pollRealKV(KeyValueHeap.java:340) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap. >
Re: Merge and HMerge
On Sat, Jan 14, 2017 at 9:50 PM, Lars George wrote: > I think that makes sense. The tool with its custom code dates back to > where we had no built in version. I am all for removing all of the tools > and leave the API call only. That is the same for an admin then compared to > calling flush or split. > > No? > > Sounds good to me. St.Ack > Lars > > Sent from my iPhone > > On 15 Jan 2017, at 04:25, Stephen Jiang wrote: > > >> If you remove the util.Merge tool, how then does an operator ask for a > merge > > in its absence? > > > > We have a shell command to merge region. In the past, it calls the same > RS > > side code. I don't think there is a need to have util.Merge (even if we > > really want, we can ask this utility to call HBaseAdmin.mergeRegions, > which > > is the same path from the merge command through 'hbase shell'). > > > > Thanks > > Stephen > > > >> On Fri, Jan 13, 2017 at 11:29 PM, Stack wrote: > >> > >> On Fri, Jan 13, 2017 at 7:16 PM, Stephen Jiang > > >> wrote: > >> > >>> Revive this thread > >>> > >>> I am in the process of removing Region Server side merge (and split) > >>> transaction code in master branch; as now we have merge (and split) > >>> procedure(s) from master doing the same thing. > >>> > >>> > >> Good (Issue?) > >> > >> > >>> The Merge tool depends on RS-side merge code. I'd like to use this > >> chance > >>> to remove the util.Merge tool. This is for 2.0 and up releases only. > >>> Deprecation does not work here; as keeping the RS-side merge code would > >>> have duplicate logic in source code and make the new Assignment manager > >>> code more complicated. > >>> > >>> > >> Could util.Merge be changed to ask the Master run the merge (via AMv2)? > >> > >> If you remove the util.Merge tool, how then does an operator ask for a > >> merge in its absence? > >> > >> Thanks Stephen > >> > >> S > >> > >> > >>> Please let me know whether you have objection. > >>> > >>> Thanks > >>> Stephen > >>> > >>> PS. I could deprecated HMerge code if anyone is really using it. It > has > >>> its own logic and standalone (supposed to dangerously work offline and > >>> merge more than 2 regions - the util.Merge and shell not support these > >>> functionality for now). > >>> > >>> On Wed, Nov 16, 2016 at 11:04 AM, Enis Söztutar > >>> wrote: > >>> > @Appy what is not clear from above? > > I think we should get rid of both Merge and HMerge. > > We should not have any tool which will work in offline mode by going > >> over > the HDFS data. Seems very brittle to be broken when things get > changed. > Only use case I can think of is that somehow you end up with a lot of > regions and you cannot bring the cluster back up because of OOMs, etc > >> and > you have to reduce the number of regions in offline mode. However, we > >> did > not see this kind of thing in any of our customers for the last couple > >> of > years so far. > > I think we should seriously look into improving normalizer and > enabling > that by default for all the tables. Ideally, normalizer should be > >> running > much more frequently, and should be configured with higher-level goals > >>> and > heuristics. Like on average how many regions per node, etc and should > >> be > looking at the global state (like the balancer) to decide on split / > >>> merge > points. > > Enis > > On Wed, Nov 16, 2016 at 1:17 AM, Apekshit Sharma > wrote: > > > bq. HMerge can merge multiple regions by going over the list of > > regions and checking > > their sizes. > > bq. But both of these tools (Merge and HMerge) are very dangerous > > > > I came across HMerge and it looks like dead code. Isn't referenced > >> from > > anywhere except one test. (This is what lars also pointed out in the > first > > email too). > > It would make perfect sense if it was a tool or was being referenced > >>> from > > somewhere, but with lack of either of that, am a bit confused here. > > @Enis, you seem to know everything about them, please educate me. > > Thanks > > - Appy > > > > > > > > On Thu, Sep 29, 2016 at 12:43 AM, Enis Söztutar > > wrote: > > > >> Merge has very limited usability singe it can do a single merge and > >>> can > >> only run when HBase is offline. > >> HMerge can merge multiple regions by going over the list of regions > >>> and > >> checking their sizes. > >> And of course we have the "supported" online merge which is the > >> shell > >> command. > >> > >> But both of these tools (Merge and HMerge) are very dangerous I > >>> think. > I > >> would say we should deprecate both to be replaced by the online > >>> merger > >> tool. We should not allow offline merge at all. I fail to see the > usecase > >> that you have to use an offline merge. > >> > >> Enis > >> > >> On Wed
Re: [VOTE] The 1st HBase 1.3.0 release candidate (RC0) is available
Late +1 Checked signature and hash on src tgz. Built tgz from src w/ jdk7. Loaded some data. Verified it made it. Browsed UI, logs, and docs. All looks good. St.Ack P.S. Sorry, didn't get a chance to deploy and run RC test at larger scale. Will do for 1.3.1. Have done a bunch of larger scale ITBLLs w/ Chaos on recent tips of 1.3 and it seemed sound. On Fri, Jan 6, 2017 at 12:42 PM, Mikhail Antonov wrote: > Hello everyone, > > I'm pleased to announce the first release candidate for HBase 1.3.0 is > available to download and testing. > > Artifacts are available here: > > https://dist.apache.org/repos/dist/dev/hbase/1.3.0RC0/ > > Maven artifacts are available in the staging repository: > > https://repository.apache.org/content/repositories/orgapachehbase-1162/ > > All artifacts are signed with my code signing key 35A4ABE2, which is also > in the project KEYS file at > > http://www.apache.org/dist/hbase/KEYS > > these artifacts correspond to commit hash > > e359c76e8d9fd0d67396456f92bcbad9ecd7a710 tagged as 1.3.0RC0. > > HBase 1.3.0 is the third minor release in the HBase 1.x line and includes > approximately 1700 resolved issues. > > Notable new features include: > > - Date-based tiered compactions (HBASE-15181, HBASE-15339) > - Maven archetypes for HBase client applications (HBASE-14877) > - Throughput controller for flushes (HBASE-14969) > - Controlled delay (CoDel) based RPC scheduler (HBASE-15136) > - Bulk loaded HFile replication (HBASE-13153) > - More improvements to Procedure V2 > - Improvements to Multi WAL (HBASE-14457) > - Many improvements and optimizations in metrics subsystem > - Reduced memory allocation in RPC layer (HBASE-15177) > - Region location lookups optimizations in HBase client > > and numerous bugfixes and performance improvements. > > The full list of issues addressed is available at > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje > ctId=12310753&version=12332794 > > and also in the CHANGES.txt file in the root directory of the source > tarball. > > Please take a few minutes to verify the release and vote on > releasing it: > > [ ] +1 Release this package as Apache HBase 1.3.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because... > > This Vote will run for one week and close Fri Jan 13, 2017 22:00 PST. > > Thank you! > Mikhail >
Re: [VOTE] The 1st HBase 1.3.0 release candidate (RC0) is available
Hi, Mikhail It looks like the hadoop 2.5.1 bundled with the HBase 1.3.0 would make the HBase servers (master and region server) fail/abort in a Kerberos environment after the initial server login expires. It happens on Java 1.8. This seems to be the related JIRA: https://issues.apache.org/jira/browse/HADOOP-10786 The root problem is UGI#reloginFromKeytab. It happened on my HBase 1.3.0 RC0 testing cluster with OpenJDk 1.8.0_101-b13 after 24 hours. Since Java 1.8 is supported by HBase 1.3.0. I think we probably should call it out in a documentation. Thanks. Jerry On Mon, Jan 16, 2017 at 2:30 AM, Mikhail Antonov wrote: > With 4 binding +1 (including mine), 2 non-binding +1 and no other votes > this vote passes and RC0 becomes 1.3.0 release. > > Thanks for everyone who tested the RC and voted! > > I'll publish artifacts and make an announcements soon, > > Thanks! > Mikhail > > > On Sat, Jan 14, 2017 at 12:12 PM, Jerry He wrote: > > > +1 > > > > - Downloaded the src tarball. > > - Checked the md5 and signature. All good. > > - Built from source. OpenJDk 1.8.0_101-b13 > > - Unit test suite. Passed. > > - mvn apache-rat:check Passed > > > > - Download the bin tarball. > > - Checked the md5 and signature. All good. > > - Untar it. Layout and files look good. > > > > - Put the binaries on a distributed Kerberos cluster > > - With all the security co-processors enabled. > > - Ran hbase shell. Table put, scan, split, major-compact, merge-region. > > Permission checkes good. > > - SecureBulkLoad good. > > - Security audit trace looks good. Master and region server logs look > > good. > > > > - Check master and region server UI. Clicked on the tabs, looked at the > > tasks running, Debug dump, Configuration dump, zk dump. Looks good. > > > > - Got this exception when running PerformanceEvaluation or other MR job: > > org.codehaus.jackson.map.exc.UnrecognizedPropertyException: > Unrecognized > > field "Token" (Class > > org.apache.hadoop.yarn.api.records.timeline. > TimelineDelegationTokenRespons > > e), > > not marked as ignorable > > at [Source: N/A; line: -1, column: -1] (through reference chain: > > org.apache.hadoop.yarn.api.records.timeline. > TimelineDelegationTokenRespons > > e["Token"]) > > > >My Hadoop/Yarn cluster is 2.7.2. HBase 1.3.0 bundles 2.5.1. That may > be > > the problem. > >After copying hadoop 2.7.2 jars into hbase. The jobs run fine. > >Loaded data. > > > > On Sat, Jan 14, 2017 at 12:51 AM, Elliott Clark > wrote: > > > > > +1 Checked sigs. > > > Downloaded everything looks good. > > > License and Notice files look good. > > > > > > On Mon, Jan 9, 2017 at 7:11 PM, Andrew Purtell > > > wrote: > > > > > > > +1 > > > > > > > > Checked sums and signatures: ok > > > > Spot check LICENSE and NOTICE files: ok > > > > Built from source (7u80): ok > > > > RAT audit (7u80): ok > > > > Unit test suite passes (8u102): > > > > Loaded 1 million rows with LTT (8u102, 10 readers, 10 writers, 10 > > > updaters > > > > @ 20%): all keys verified, no unexpected or unusual messages in the > > logs, > > > > latencies in the ballpark > > > > 1 billion row ITBLL (8u102): failed, but known issue HBASE-17069 > > > > > > > > > > > > > > > > On Fri, Jan 6, 2017 at 12:42 PM, Mikhail Antonov < > olorinb...@gmail.com > > > > > > > wrote: > > > > > > > > > Hello everyone, > > > > > > > > > > I'm pleased to announce the first release candidate for HBase 1.3.0 > > is > > > > > available to download and testing. > > > > > > > > > > Artifacts are available here: > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hbase/1.3.0RC0/ > > > > > > > > > > Maven artifacts are available in the staging repository: > > > > > > > > > > https://repository.apache.org/content/repositories/ > > > orgapachehbase-1162/ > > > > > > > > > > All artifacts are signed with my code signing key 35A4ABE2, which > is > > > also > > > > > in the project KEYS file at > > > > > > > > > > http://www.apache.org/dist/hbase/KEYS > > > > > > > > > > these artifacts correspond to commit hash > > > > > > > > > > e359c76e8d9fd0d67396456f92bcbad9ecd7a710 tagged as 1.3.0RC0. > > > > > > > > > > HBase 1.3.0 is the third minor release in the HBase 1.x line and > > > includes > > > > > approximately 1700 resolved issues. > > > > > > > > > > Notable new features include: > > > > > > > > > > - Date-based tiered compactions (HBASE-15181, HBASE-15339) > > > > > - Maven archetypes for HBase client applications (HBASE-14877) > > > > > - Throughput controller for flushes (HBASE-14969) > > > > > - Controlled delay (CoDel) based RPC scheduler (HBASE-15136) > > > > > - Bulk loaded HFile replication (HBASE-13153) > > > > > - More improvements to Procedure V2 > > > > > - Improvements to Multi WAL (HBASE-14457) > > > > > - Many improvements and optimizations in metrics subsystem > > > > > - Reduced memory allocation in RPC layer (HBASE-15177) > > > > > - Region location lookups optimizations in HBase client > > > >
Re: Resignation as RM for branch 0.98
Thanks for your efforts Andrew. +1 on EOL of 0.98. On Mon, Jan 16, 2017 at 10:27 PM, Josh Elser wrote: > Big +1 to your efforts, Andrew, and an EOL on 0.98 > > > Ted Yu wrote: > >> Kudos to Andrew. >> >> On Mon, Jan 9, 2017 at 11:26 AM, Enis Söztutar wrote: >> >> Thanks Andrew for carrying this. >>> >>> +1 on the EOL messaging. >>> >>> Enis >>> >>> On Mon, Jan 9, 2017 at 10:37 AM, Mikhail Antonov >>> wrote: >>> >>> Yeah - thank you for all hard work and time investment on reviews, cherry-picking, backporting and testing Andrew! +1 to retire 0.98. -Mikhail On Mon, Jan 9, 2017 at 10:28 AM, Sean Busbey wrote: Thanks for all the hard work around 0.98 Andrew! > > I'm +1 on retiring 0.98, but I think we should solicit feedback from > user@hbase before making a decision. > > On Fri, Jan 6, 2017 at 6:51 PM, Andrew Purtell > wrote: > >> HBasers, >> >> Happy new year! >> >> The 0.98 branch has had a great run at a fairly constant release >> > cadence, > producing 24 releases in all. >> >> I served as the branch RM for 98 for all of that time, and now I feel >> > the > time has come to move on to work with more recent code. Therefore >> > please > consider this my resignation as such (it's an informal position after >> > all > (smile)) from the role of branch 98 RM. >> >> It will be up to the PMC to decide what to do about 98 going forward. >> > My > recommendation is to put out the message that 98 branch has been end >> > of >>> lifed (EOL). There is no reason someone else couldn't don the RM hat >> > and > put out an additional 98 release, for example, in order to deal with >> > a >>> critical fix for something we just can't abide shipping as the last >> > release > >> of anything. That person might even be me. However, without a >> > dedicated >>> shepard the management of the 98 branch is pretty difficult these >> > days. >>> It > >> requires Java 6 to build, which requires a hack to its security >> > classes >>> to > >> communicate with modern SSL servers like repository.apache.org, and >> mandates Hadoop 1 compatibility, which can lead to tricky problems >> > found > only at release time. Backports are rarely clean. Assume 30 minutes >> > to >>> 2 > hours work for each attempt at a backport, including time for tests >> > to >>> complete. Maintaining and releasing 98 branch will be a time >> > commitment. > FWIW. All that said if someone else would like to step up into the 98 >> > RM > role, then we don't need to consider calling it EOL. However, as I've >> > said, > >> that is my humble recommendation to the PMC. >> >> >> -- >> Best regards, >> >> - Andy >> >> If you are given a choice, you believe you have acted freely. - >> > Raymond >>> Teller (via Peter Watts) >> > -- Thanks, Michael Antonov >>
Successful: hbase.apache.org HTML Checker
Successful If successful, the HTML and link-checking report for http://hbase.apache.org is available at https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/78/artifact/link_report/index.html. If failed, see https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/78/console.
[jira] [Created] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
Allan Yang created HBASE-17475:
--
Summary: Stack overflow in AsyncProcess if retry too much
Key: HBASE-17475
URL: https://issues.apache.org/jira/browse/HBASE-17475
Project: HBase
Issue Type: Bug
Components: API
Affects Versions: 2.0.0, 1.4.0
Reporter: Allan Yang
Assignee: Allan Yang
In AsyncProcess, we resubmit the retry task in the same thread
{code}
// run all the runnables
for (Runnable runnable : runnables) {
if ((--actionsRemaining == 0) && reuseThread) {
runnable.run();
} else {
try {
pool.submit(runnable);
}
..
{code}
But, if we retry too much time. soon, stack overflow will occur. This is very
common in clusters with Phoenix. Phoenix need to write index table in the
normal write path, retry will cause stack overflow exception.
{noformat}
"htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
nid=0x4a96 waiting on condition [0x7fe3f6301000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
at
org.apache.hadoop.hbase.client.Asy
Re: Moving 2.0 forward
On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang wrote: > For 6. Significant contirbs in master only, there are some issues about > replication operations routed through master. They are sub-task > of HBASE-10504. And there are other umbrella issue for replication, like > HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication > tracking from Zookeeper to HBase. So i thought we can add a new section > named > hbase-replication to possible 2.0.0s. This will help us to track the state. > Thanks. > Thanks Guanghao Zhang. I agree. I made you an editor. If you want to have a go at a first cut, be my guest. If nothing done in the next day or so, I'll add this section Sir. Thanks, M
[jira] [Created] (HBASE-17474) Reduce frequency of NoSuchMethodException when calling setStoragePolicy()
Ted Yu created HBASE-17474:
--
Summary: Reduce frequency of NoSuchMethodException when calling
setStoragePolicy()
Key: HBASE-17474
URL: https://issues.apache.org/jira/browse/HBASE-17474
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
Saw the following in test output:
{code}
2017-01-15 09:37:53,383 WARN [StoreOpener-98f4c892b5bcd9188f06331426e710f9-1]
regionserver.HRegionFileSystem(199): Failed to set storage policy of
[/Users/tyu/trunk/hbase-
server/target/test-data/e052b6ad-02e2-41bf-a806-d617900a8c22/TestStoreFileRefresherChore/data/default/testIsStale/98f4c892b5bcd9188f06331426e710f9/cf]
to [HOT]
java.lang.UnsupportedOperationException: Cannot find specified method
setStoragePolicy
at
org.apache.hadoop.hbase.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:209)
at
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:197)
at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:244)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5254)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:988)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:985)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy(org.apache.hadoop.fs.Path,
java.lang.String)
at java.lang.Class.getMethod(Class.java:1786)
at
org.apache.hadoop.hbase.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:205)
... 11 more
{code}
setStoragePolicy() is not implemented by LocalFileSystem.
We should reduce the frequency of the above log since it clutters test output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-17461) HBase shell "major_compact" command should properly convert "table_or_region_name" parameter to java byte array properly before simply calling "HBaseAdmin.majorCompact"
[
https://issues.apache.org/jira/browse/HBASE-17461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil resolved HBASE-17461.
--
Resolution: Invalid
> HBase shell "major_compact" command should properly convert
> "table_or_region_name" parameter to java byte array properly before simply
> calling "HBaseAdmin.majorCompact" method
> ---
>
> Key: HBASE-17461
> URL: https://issues.apache.org/jira/browse/HBASE-17461
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>
> On HBase shell, *major_compact* command simply passes the received
> *table_or_region_name* parameter straight to java *HBaseAdmin.majorCompact*
> method.
> On some corner cases, HBase tables row keys may have special characters.
> Then, if a region is split in such a way that row keys with special
> characters are now part of the region name, calling *major_compact* on this
> regions will fail, if the special character ASCII code is higher than 127.
> This happens because Java byte type is signed, while ruby byte type isn't,
> causing the region name to be converted to a wrong string at Java side.
> For example, considering a region named as below:
> {noformat}
> test,\xF8\xB9B2!$\x9C\x0A\xFEG\xC0\xE3\x8B\x1B\xFF\x15,1481745228583.b4bc69356d89018bfad3ee106b717285.
> {noformat}
> Calling major_compat on it fails as follows:
> {noformat}
> hbase(main):008:0* major_compact
> "test,\xF8\xB9B2!$\x9C\x0A\xFEG\xC0\xE3\x8B\x1B\xFF\x15,1484177359169.8128fa75ae0cd4eba38da2667ac8ec98."
> ERROR: Illegal character code:44, <,> at 4. User-space table qualifiers can
> only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9-.]: test,�B2!$�
> �G���1484177359169.8128fa75ae0cd4eba38da2667ac8ec98.
> {noformat}
> An easy solution is to convert *table_or_region_name* parameter properly,
> prior to calling *HBaseAdmin.majorCompact* in the same way as it's already
> done on some other shell commands, such as *get*:
> {noformat}
> admin.major_compact(table_or_region_name.to_s.to_java_bytes, family)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Resignation as RM for branch 0.98
Big +1 to your efforts, Andrew, and an EOL on 0.98 Ted Yu wrote: Kudos to Andrew. On Mon, Jan 9, 2017 at 11:26 AM, Enis Söztutar wrote: Thanks Andrew for carrying this. +1 on the EOL messaging. Enis On Mon, Jan 9, 2017 at 10:37 AM, Mikhail Antonov wrote: Yeah - thank you for all hard work and time investment on reviews, cherry-picking, backporting and testing Andrew! +1 to retire 0.98. -Mikhail On Mon, Jan 9, 2017 at 10:28 AM, Sean Busbey wrote: Thanks for all the hard work around 0.98 Andrew! I'm +1 on retiring 0.98, but I think we should solicit feedback from user@hbase before making a decision. On Fri, Jan 6, 2017 at 6:51 PM, Andrew Purtell wrote: HBasers, Happy new year! The 0.98 branch has had a great run at a fairly constant release cadence, producing 24 releases in all. I served as the branch RM for 98 for all of that time, and now I feel the time has come to move on to work with more recent code. Therefore please consider this my resignation as such (it's an informal position after all (smile)) from the role of branch 98 RM. It will be up to the PMC to decide what to do about 98 going forward. My recommendation is to put out the message that 98 branch has been end of lifed (EOL). There is no reason someone else couldn't don the RM hat and put out an additional 98 release, for example, in order to deal with a critical fix for something we just can't abide shipping as the last release of anything. That person might even be me. However, without a dedicated shepard the management of the 98 branch is pretty difficult these days. It requires Java 6 to build, which requires a hack to its security classes to communicate with modern SSL servers like repository.apache.org, and mandates Hadoop 1 compatibility, which can lead to tricky problems found only at release time. Backports are rarely clean. Assume 30 minutes to 2 hours work for each attempt at a backport, including time for tests to complete. Maintaining and releasing 98 branch will be a time commitment. FWIW. All that said if someone else would like to step up into the 98 RM role, then we don't need to consider calling it EOL. However, as I've said, that is my humble recommendation to the PMC. -- Best regards, - Andy If you are given a choice, you believe you have acted freely. - Raymond Teller (via Peter Watts) -- Thanks, Michael Antonov
Re: Region comapction failed
This is likely a new bug. Please consider opening a JIRA. > On Jan 16, 2017, at 2:22 AM, Pankaj kr wrote: > > Yeah, HBASE-13123 already available. > > While going through HFilePrettyPrinter result with -e option, we observed > some keys doesn't contain complete column_family:qualifier information. > Normally in HFIle Keys are formed as, >K: r1/CF:Qualifier/1484302946516/Put/vlen=1/seqid=0 > > But some records, >K: r1/Unknown_Qualifier/1484302946516/Put/vlen=1/seqid=0 > Here there is no column family detail, normally it is like column family name > and the colon (:). > > It's strange, how key is written in wrong format in HFile. > > > Regards, > Pankaj > > -Original Message- > From: Ted Yu [mailto:yuzhih...@gmail.com] > Sent: Saturday, January 14, 2017 9:41 AM > To: dev@hbase.apache.org > Cc: u...@hbase.apache.org > Subject: Re: Region comapction failed > > w.r.t. #2, I did a quick search for bloom related fixes. > > I found HBASE-13123 but it was in 1.0.2 > > Planning to spend more time in the next few days. > >> On Fri, Jan 13, 2017 at 5:29 PM, Pankaj kr wrote: >> >> Thanks Ted for replying. >> >> Actually issue happened in production environment and there are many >> HFiles in that store (can't get the file). As we don't log the file >> name which is corrupted, Is there anyway to get the corrupted file name? >> >> Block encoding is "NONE", table schema has bloom filter as "ROW", >> compression type is "Snappy" and durability is SKIP_WAL. >> >> >> Regards, >> Pankaj >> >> >> -Original Message- >> From: Ted Yu [mailto:yuzhih...@gmail.com] >> Sent: Friday, January 13, 2017 10:30 PM >> To: dev@hbase.apache.org >> Cc: u...@hbase.apache.org >> Subject: Re: Region comapction failed >> >> In the second case, the error happened when writing hfile. Can you >> track down the path of the new file so that further investigation can be >> done ? >> >> Does the table use any encoding ? >> >> Thanks >> >>> On Jan 13, 2017, at 2:47 AM, Pankaj kr wrote: >>> >>> Hi, >>> >>> We met a weird issue in our production environment. >>> >>> Region compaction is always failing with following errors, >>> >>> 1. >>> 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP: >> PORT-longCompactions-1483858654825 | Compaction failed Request = >> regionName=., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M, >> 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5, >> time=1747414906352088 | >> org.apache.hadoop.hbase.regionserver.CompactSplitThread$ >> CompactionRunner.doCompaction(CompactSplitThread.java:562) >>> java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into >>> a >> column actually smaller than the previous column: XXX >>> at org.apache.hadoop.hbase.regionserver. >> ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker. >> java:114) >>> at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher. >> match(ScanQueryMatcher.java:457) >>> at org.apache.hadoop.hbase.regionserver.StoreScanner. >> next(StoreScanner.java:551) >>> at org.apache.hadoop.hbase.regionserver.compactions. >> Compactor.performCompaction(Compactor.java:328) >>> at org.apache.hadoop.hbase.regionserver.compactions. >> DefaultCompactor.compact(DefaultCompactor.java:104) >>> at org.apache.hadoop.hbase.regionserver. >> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine. >> java:133) >>> at >>> org.apache.hadoop.hbase.regionserver.HStore.compact( >> HStore.java:1243) >>> at >>> org.apache.hadoop.hbase.regionserver.HRegion.compact( >> HRegion.java:1895) >>> at org.apache.hadoop.hbase.regionserver. >> CompactSplitThread$CompactionRunner.doCompaction( >> CompactSplitThread.java:546) >>> at org.apache.hadoop.hbase.regionserver. >> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583) >>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1142) >>> at java.util.concurrent.ThreadPoolExecuto >>> >>> 2. >>> 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP: >> PORT-longCompactions-1483686810953 | Compaction failed Request = >> regionName=YY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9 >> M, >> 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68, >> time=1748294500157323 | org.apache.hadoop.hbase.regionserver. >> CompactSplitThread$CompactionRunner.doCompaction( >> CompactSplitThread.java:562) >>> java.io.IOException: Non-increasing Bloom keys: >>> XX >> after >>> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer. >> appendGeneralBloomfilter(StoreFile.java:911) >>> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer. >> append(StoreFile.java:947) >>> at org.apache.hadoop.hbase.regionserver.compactions. >> Compactor.pe
Re: Moving 2.0 forward
For 6. Significant contirbs in master only, there are some issues about replication operations routed through master. They are sub-task of HBASE-10504. And there are other umbrella issue for replication, like HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication tracking from Zookeeper to HBase. So i thought we can add a new section named hbase-replication to possible 2.0.0s. This will help us to track the state. Thanks.
Re: [VOTE] The 1st HBase 1.3.0 release candidate (RC0) is available
With 4 binding +1 (including mine), 2 non-binding +1 and no other votes this vote passes and RC0 becomes 1.3.0 release. Thanks for everyone who tested the RC and voted! I'll publish artifacts and make an announcements soon, Thanks! Mikhail On Sat, Jan 14, 2017 at 12:12 PM, Jerry He wrote: > +1 > > - Downloaded the src tarball. > - Checked the md5 and signature. All good. > - Built from source. OpenJDk 1.8.0_101-b13 > - Unit test suite. Passed. > - mvn apache-rat:check Passed > > - Download the bin tarball. > - Checked the md5 and signature. All good. > - Untar it. Layout and files look good. > > - Put the binaries on a distributed Kerberos cluster > - With all the security co-processors enabled. > - Ran hbase shell. Table put, scan, split, major-compact, merge-region. > Permission checkes good. > - SecureBulkLoad good. > - Security audit trace looks good. Master and region server logs look > good. > > - Check master and region server UI. Clicked on the tabs, looked at the > tasks running, Debug dump, Configuration dump, zk dump. Looks good. > > - Got this exception when running PerformanceEvaluation or other MR job: > org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized > field "Token" (Class > org.apache.hadoop.yarn.api.records.timeline.TimelineDelegationTokenRespons > e), > not marked as ignorable > at [Source: N/A; line: -1, column: -1] (through reference chain: > org.apache.hadoop.yarn.api.records.timeline.TimelineDelegationTokenRespons > e["Token"]) > >My Hadoop/Yarn cluster is 2.7.2. HBase 1.3.0 bundles 2.5.1. That may be > the problem. >After copying hadoop 2.7.2 jars into hbase. The jobs run fine. >Loaded data. > > On Sat, Jan 14, 2017 at 12:51 AM, Elliott Clark wrote: > > > +1 Checked sigs. > > Downloaded everything looks good. > > License and Notice files look good. > > > > On Mon, Jan 9, 2017 at 7:11 PM, Andrew Purtell > > wrote: > > > > > +1 > > > > > > Checked sums and signatures: ok > > > Spot check LICENSE and NOTICE files: ok > > > Built from source (7u80): ok > > > RAT audit (7u80): ok > > > Unit test suite passes (8u102): > > > Loaded 1 million rows with LTT (8u102, 10 readers, 10 writers, 10 > > updaters > > > @ 20%): all keys verified, no unexpected or unusual messages in the > logs, > > > latencies in the ballpark > > > 1 billion row ITBLL (8u102): failed, but known issue HBASE-17069 > > > > > > > > > > > > On Fri, Jan 6, 2017 at 12:42 PM, Mikhail Antonov > > > > wrote: > > > > > > > Hello everyone, > > > > > > > > I'm pleased to announce the first release candidate for HBase 1.3.0 > is > > > > available to download and testing. > > > > > > > > Artifacts are available here: > > > > > > > > https://dist.apache.org/repos/dist/dev/hbase/1.3.0RC0/ > > > > > > > > Maven artifacts are available in the staging repository: > > > > > > > > https://repository.apache.org/content/repositories/ > > orgapachehbase-1162/ > > > > > > > > All artifacts are signed with my code signing key 35A4ABE2, which is > > also > > > > in the project KEYS file at > > > > > > > > http://www.apache.org/dist/hbase/KEYS > > > > > > > > these artifacts correspond to commit hash > > > > > > > > e359c76e8d9fd0d67396456f92bcbad9ecd7a710 tagged as 1.3.0RC0. > > > > > > > > HBase 1.3.0 is the third minor release in the HBase 1.x line and > > includes > > > > approximately 1700 resolved issues. > > > > > > > > Notable new features include: > > > > > > > > - Date-based tiered compactions (HBASE-15181, HBASE-15339) > > > > - Maven archetypes for HBase client applications (HBASE-14877) > > > > - Throughput controller for flushes (HBASE-14969) > > > > - Controlled delay (CoDel) based RPC scheduler (HBASE-15136) > > > > - Bulk loaded HFile replication (HBASE-13153) > > > > - More improvements to Procedure V2 > > > > - Improvements to Multi WAL (HBASE-14457) > > > > - Many improvements and optimizations in metrics subsystem > > > > - Reduced memory allocation in RPC layer (HBASE-15177) > > > > - Region location lookups optimizations in HBase client > > > > > > > > and numerous bugfixes and performance improvements. > > > > > > > > The full list of issues addressed is available at > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > projectId=12310753&version=12332794 > > > > > > > > and also in the CHANGES.txt file in the root directory of the source > > > > tarball. > > > > > > > > Please take a few minutes to verify the release and vote on > > > > releasing it: > > > > > > > > [ ] +1 Release this package as Apache HBase 1.3.0 > > > > [ ] +0 no opinion > > > > [ ] -1 Do not release this package because... > > > > > > > > This Vote will run for one week and close Fri Jan 13, 2017 22:00 PST. > > > > > > > > Thank you! > > > > Mikhail > > > > > > > > > > > > > > > > -- > > > Best regards, > > > > > >- Andy > > > > > > If you are given a choice, you believe you have acted freely. - Raymond > > > Teller (via Peter Watts) >
[jira] [Created] (HBASE-17473) Set capacity on ArrayList in hbase-client
Jan Hentschel created HBASE-17473: - Summary: Set capacity on ArrayList in hbase-client Key: HBASE-17473 URL: https://issues.apache.org/jira/browse/HBASE-17473 Project: HBase Issue Type: Sub-task Components: Client Reporter: Jan Hentschel Assignee: Jan Hentschel Priority: Minor Set the capacity on an ArrayList when possible in the *hbase-client* module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
RE: Region comapction failed
Yeah, HBASE-13123 already available. While going through HFilePrettyPrinter result with -e option, we observed some keys doesn't contain complete column_family:qualifier information. Normally in HFIle Keys are formed as, K: r1/CF:Qualifier/1484302946516/Put/vlen=1/seqid=0 But some records, K: r1/Unknown_Qualifier/1484302946516/Put/vlen=1/seqid=0 Here there is no column family detail, normally it is like column family name and the colon (:) . It's strange, how key is written in wrong format in HFile. Regards, Pankaj -Original Message- From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Saturday, January 14, 2017 9:41 AM To: dev@hbase.apache.org Cc: u...@hbase.apache.org Subject: Re: Region comapction failed w.r.t. #2, I did a quick search for bloom related fixes. I found HBASE-13123 but it was in 1.0.2 Planning to spend more time in the next few days. On Fri, Jan 13, 2017 at 5:29 PM, Pankaj kr wrote: > Thanks Ted for replying. > > Actually issue happened in production environment and there are many > HFiles in that store (can't get the file). As we don't log the file > name which is corrupted, Is there anyway to get the corrupted file name? > > Block encoding is "NONE", table schema has bloom filter as "ROW", > compression type is "Snappy" and durability is SKIP_WAL. > > > Regards, > Pankaj > > > -Original Message- > From: Ted Yu [mailto:yuzhih...@gmail.com] > Sent: Friday, January 13, 2017 10:30 PM > To: dev@hbase.apache.org > Cc: u...@hbase.apache.org > Subject: Re: Region comapction failed > > In the second case, the error happened when writing hfile. Can you > track down the path of the new file so that further investigation can be done > ? > > Does the table use any encoding ? > > Thanks > > > On Jan 13, 2017, at 2:47 AM, Pankaj kr wrote: > > > > Hi, > > > > We met a weird issue in our production environment. > > > > Region compaction is always failing with following errors, > > > > 1. > > 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP: > PORT-longCompactions-1483858654825 | Compaction failed Request = > regionName=., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M, > 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5, > time=1747414906352088 | > org.apache.hadoop.hbase.regionserver.CompactSplitThread$ > CompactionRunner.doCompaction(CompactSplitThread.java:562) > > java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into > > a > column actually smaller than the previous column: XXX > >at org.apache.hadoop.hbase.regionserver. > ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker. > java:114) > >at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher. > match(ScanQueryMatcher.java:457) > >at org.apache.hadoop.hbase.regionserver.StoreScanner. > next(StoreScanner.java:551) > >at org.apache.hadoop.hbase.regionserver.compactions. > Compactor.performCompaction(Compactor.java:328) > >at org.apache.hadoop.hbase.regionserver.compactions. > DefaultCompactor.compact(DefaultCompactor.java:104) > >at org.apache.hadoop.hbase.regionserver. > DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine. > java:133) > >at > > org.apache.hadoop.hbase.regionserver.HStore.compact( > HStore.java:1243) > >at > > org.apache.hadoop.hbase.regionserver.HRegion.compact( > HRegion.java:1895) > >at org.apache.hadoop.hbase.regionserver. > CompactSplitThread$CompactionRunner.doCompaction( > CompactSplitThread.java:546) > >at org.apache.hadoop.hbase.regionserver. > CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583) > >at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > >at java.util.concurrent.ThreadPoolExecuto > > > > 2. > > 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP: > PORT-longCompactions-1483686810953 | Compaction failed Request = > regionName=YY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9 > M, > 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68, > time=1748294500157323 | org.apache.hadoop.hbase.regionserver. > CompactSplitThread$CompactionRunner.doCompaction( > CompactSplitThread.java:562) > > java.io.IOException: Non-increasing Bloom keys: > > XX > after > >at org.apache.hadoop.hbase.regionserver.StoreFile$Writer. > appendGeneralBloomfilter(StoreFile.java:911) > >at org.apache.hadoop.hbase.regionserver.StoreFile$Writer. > append(StoreFile.java:947) > >at org.apache.hadoop.hbase.regionserver.compactions. > Compactor.performCompaction(Compactor.java:337) > >at org.apache.hadoop.hbase.regionserver.compactions. > DefaultCompactor.compact(DefaultCompactor.java:104) > >at org.apache.hadoop.hbase.regionserver. > DefaultStor
