[ANNOUNCE] New HBase committer Andor Molnár

2024-05-29 Thread Duo Zhang
On behalf of the Apache HBase PMC, I am pleased to announce that Andor Molnár(andor) has accepted the PMC's invitation to become a committer on the project. We appreciate all of Andor Molnár's generous contributions thus far and look forward to his continued involvement. Congratulations and

[ANNOUNCE] New HBase committer Andor Molnár

2024-05-29 Thread Duo Zhang
On behalf of the Apache HBase PMC, I am pleased to announce that Andor Molnár(andor) has accepted the PMC's invitation to become a committer on the project. We appreciate all of Andor Molnár's generous contributions thus far and look forward to his continued involvement. Congratulations and

[ANNOUNCE] Apache HBase 2.4.18 is now available for download

2024-05-25 Thread Duo Zhang
The HBase team is happy to announce the immediate availability of HBase 2.4.18. Apache HBase™ is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To learn

[ANNOUNCE] Apache HBase 2.4.18 is now available for download

2024-05-25 Thread Duo Zhang
The HBase team is happy to announce the immediate availability of HBase 2.4.18. Apache HBase™ is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To learn

Re: [ANNOUNCE] Apache HBase 2.6.0 is now available for download

2024-05-20 Thread Duo Zhang
Congratulations! But it seems you missed the replacement for the first '_version_' placeholder... Bryan Beaudreault 于2024年5月21日周二 00:44写道: > > The HBase team is happy to announce the immediate availability of HBase > _version_. > > Apache HBase™ is an open-source, distributed, versioned,

Re: [ANNOUNCE] Apache HBase 2.6.0 is now available for download

2024-05-20 Thread Duo Zhang
Congratulations! But it seems you missed the replacement for the first '_version_' placeholder... Bryan Beaudreault 于2024年5月21日周二 00:44写道: > > The HBase team is happy to announce the immediate availability of HBase > _version_. > > Apache HBase™ is an open-source, distributed, versioned,

HBase Quarterly report Jan-Mar 2024

2024-05-18 Thread Duo Zhang
Hi all, HBase submits a report to the ASF board once a quarter, to inform the board about project health. I'm sending the report to the user@ and dev@ mailing lists because you are the project, and for transparency. If you have any questions about the report or the running of the project, you can

HBase Quarterly report Jan-Mar 2024

2024-05-18 Thread Duo Zhang
Hi all, HBase submits a report to the ASF board once a quarter, to inform the board about project health. I'm sending the report to the user@ and dev@ mailing lists because you are the project, and for transparency. If you have any questions about the report or the running of the project, you can

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-05-04 Thread Duo Zhang
t; Best regards > Udo > > > > > Am 30.04.2024 um 04:42 schrieb 张铎(Duo Zhang) : > > > > Oh, there is a typo, I mean the ServerCrashProcedure should not block other > > procedures if it is in claim replication queue stage. > > > > 张铎(Duo Zhang) 于202

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-29 Thread Duo Zhang
Oh, there is a typo, I mean the ServerCrashProcedure should not block other procedures if it is in claim replication queue stage. 张铎(Duo Zhang) 于2024年4月30日 周二10:41写道: > Sorry to be a pain as the procedure store is a big problem before HBase > 2.3 so we have done a big refactoring on HBa

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-29 Thread Duo Zhang
hard to upgrade the cluster? Is it worthwhile to take the > next stable version 2.5.8? > And - btw- what is the difference between the two distributions „bin“ and > „hadoop3-bin“? > > Best regards > Udo > > > > > > > Am 28.04.2024 um 03:03 schrieb 张铎(Duo Zhan

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-23 Thread Duo Zhang
master which is hanging? It seems that you are connecting the backup master... Thanks. 张铎(Duo Zhang) 于2024年4月23日周二 15:31写道: > > Ah, NPE usually means a code bug, then there is no simple way to fix > it, need to take a deep look on the code :( > > Sorry. > > Udo Offermann

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-23 Thread Duo Zhang
ty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) > at > org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) > at > org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) > at > o

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-21 Thread Duo Zhang
e“. > BTW, is it ok to delete the zookeeper directory? > > Best regards > Udo > > > > Am 20.04.2024 um 15:53 schrieb 张铎(Duo Zhang) : > > > > OK, it was waitForMetaOnline. > > > > Maybe the problem is that you do have some correct proced

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-20 Thread Duo Zhang
OK, it was waitForMetaOnline. Maybe the problem is that you do have some correct procedures before upgrading, like ServerCrashProcedure, but then you delete all the procedure wals so the ServerCrashProcedure is also gone, so meta can never be online. Please check the /hbase/meta-region-server

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-20 Thread Duo Zhang
Just post it somewhere so we can check it. Udo Offermann 于2024年4月20日周六 20:25写道: > > I do have the dump File from the web ui. I can sende it all or you Tell me > threads you are interessted in. Fortunately they all have meaningfull named. > > 张铎(Duo Zhang) schrieb am Sa., 20. A

Re: HBase Master hangs on startup during upgrade from 2.2.5 to 2.5.7

2024-04-20 Thread Duo Zhang
What is the jstack result for HMaster while hanging? Wait on the namespace table online or meta table online? Udo Offermann 于2024年4月20日周六 19:43写道: > > Hello everyone, > > We are upgrading our Hadoop/HBase cluster from Hadoop 2.8.5 & HBase 2.2.5 > to Hadoop 3.3.6 & HBase 2.5.7 > > The Hadoop

Re: Considering deprecation and removal of XZ compression (hbase-compression-xz)

2024-04-02 Thread Duo Zhang
For me I've never seen people actually use the xz compression. For size, usually people will choose gzip, and for speed, in the past people will choose lzo and now they choose snappy or zstd. So for me I prefer we just deprecated the xz compression immediately and remove it 2.6.0. Thanks.

[ANNOUNCE] New HBase committer Istvan Toth

2024-04-02 Thread Duo Zhang
On behalf of the Apache HBase PMC, I am pleased to announce that Istvan Toth(stoty) has accepted the PMC's invitation to become a committer on the project. We appreciate all of Istvan Toth's generous contributions thus far and look forward to his continued involvement. Congratulations and

[ANNOUNCE] New HBase committer Istvan Toth

2024-04-02 Thread Duo Zhang
On behalf of the Apache HBase PMC, I am pleased to announce that Istvan Toth(stoty) has accepted the PMC's invitation to become a committer on the project. We appreciate all of Istvan Toth's generous contributions thus far and look forward to his continued involvement. Congratulations and

Re: Rolling upgrade from HBase 2.2.2 to 2.5.8 [typo corrected]: There are 2336 corrupted procedures

2024-03-25 Thread Duo Zhang
OK, glad to hear that. Hope the rolling upgrading goes well Adam Sjøgren 于2024年3月25日周一 21:34写道: > > 张铎(Duo Zhang) writes: > > > Is it OK if you just restart with the 2.2.2 master? > > Yes, it was. > > After finding this StackOverflow question: > https://stackoverflo

Re: Rolling upgrade from HBase 2.2.2 to 2.5.4: There are 2336 corrupted procedures

2024-03-25 Thread Duo Zhang
Is it OK if you just restart with the 2.2.2 master? Adam Sjøgren 于2024年3月25日周一 20:14写道: > > Hi, > > I am trying to do a rolling upgrade of an HBase 2.2.2 cluster to > 2.5.4. > > When I try switching over to the new master, it processes the WAL > files and then ends up saying: > > [... many

HBase Quarterly report Oct-Dec 2023

2024-01-28 Thread Duo Zhang
Hi all, HBase submits a report to the ASF board once a quarter, to inform the board about project health. I'm sending the report to the user@ and dev@ mailing lists because you are the project, and for transparency. If you have any questions about the report or the running of the project, you can

HBase Quarterly report Oct-Dec 2023

2024-01-28 Thread Duo Zhang
Hi all, HBase submits a report to the ASF board once a quarter, to inform the board about project health. I'm sending the report to the user@ and dev@ mailing lists because you are the project, and for transparency. If you have any questions about the report or the running of the project, you can

[ANNOUNCE] Apache HBase 3.0.0-beta-1 is now available for download

2024-01-19 Thread Duo Zhang
The HBase team is happy to announce the immediate availability of HBase 3.0.0-beta-1. Apache HBase™ is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To

[ANNOUNCE] Apache HBase 3.0.0-beta-1 is now available for download

2024-01-19 Thread Duo Zhang
The HBase team is happy to announce the immediate availability of HBase 3.0.0-beta-1. Apache HBase™ is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To

Re: Engineering blog post on HBase/Phoenix major upgrade without downtime

2024-01-12 Thread Duo Zhang
Great article! And congratulations on the successful upgrade! Viraj Jasani 于2024年1月13日周六 08:21写道: > > Hello, > > We have completed major upgrade of HBase/Phoenix from 1.6/4.16 to 2.4/5.1 > versions and continuing journey to 2.5 release in production. Would like to > share some insights on how

Re: version of haddop in hbase binary download

2023-12-18 Thread Duo Zhang
In general, HBase will bundle the oldest hadoop binaries it could support by default. It does not mean that you should have your hadoop servers, like HDFS or YARN, has the same version. You are free to use newer version of hadoop to setup HDFS or YARN, as a hadoop client with lower version can

Re: [DISCUSS] End support for hadoop 2.10?

2023-12-09 Thread Duo Zhang
ideally isn't). > > I agree we need to support hadoop-2.x for a while, but we can keep that > support in hbase 2.5. This is how we've handled other hadoop versions > according to our compatibility matrix. > > On Wed, Dec 6, 2023 at 1:53 AM 张铎(Duo Zhang) wrote: > > > Bette

Re: [DISCUSS] End support for hadoop 2.10?

2023-12-05 Thread Duo Zhang
Better also send the email to user@hbase to see what our users think. I think we could change the default profile to hadoop3, but better still have the hadoop2 profile as there could still be users on hadoop-2.x. We will completely drop the hadoop2 support in hbase 3.x. Tak Lon (Stephen) Wu

Re: Stripe Compaction 策略遇到的一些问题请教

2023-11-28 Thread Duo Zhang
果然 https://github.com/apache/hbase/blob/4d90b918a3702b4e4ae2f9ee890c14665e821c01/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreEngine.java#L84 这个方法里压根就没用过那个 forceMajor 参数 可以开个 issue 吧,看看这块怎么改一下,至少得把 forceMajor 这个参数传递下去,在 stripe 内部做 compact 的时候需要能拿到这个参数 张铎(Duo Zhang

Re: Stripe Compaction 策略遇到的一些问题请教

2023-11-28 Thread Duo Zhang
我印象里应该是有这个机制的,是不是 StripeCompaction 里没考虑这个参数 leojie 于2023年11月21日周二 11:42写道: > > 张老师, > 您好! > 请教一个HBase的问题,我们线上有张表,应用了Stripe > Compaction策略,每个region均值40G,被分成8个Stripe,每个Stripe5g,业务删除大量数据,表整体和单个region手动触发major > compaction不起作用,挑选不出来合适的文件参与合并。 > >

Re: Undeleted replication queue for removed peer found

2023-11-18 Thread Duo Zhang
I guess the problem is you exceeded the maximum size limit for zookeeper multi operation. I searched the code base of branch-1, you could try to set 'hbase.zookeeper.useMulti' to false in your hbase-site.xml to disable multi so the operation could succeed. But it may introduce inconsistency so

Re: How to proactively update the information of hdfs blocks in hbase cache

2023-10-30 Thread Duo Zhang
See https://issues.apache.org/jira/browse/HDFS-13571. The problem for HBase is that the region server will keep a large amount of DFSInputStream open, for random access. If a DN node is completely gone, i.e, can not be reached through the network, you will face a connection timeout, usually it

[ANNOUNCE] Please welcome Bryan Beaudreault to the HBase PMC

2023-10-16 Thread Duo Zhang
On behalf of the Apache HBase PMC I am pleased to announce that Bryan Beaudreault has accepted our invitation to become a PMC member on the Apache HBase project. We appreciate Bryan Beaudreault stepping up to take more responsibility in the HBase project. Please join me in welcoming Bryan

[ANNOUNCE] Please welcome Bryan Beaudreault to the HBase PMC

2023-10-16 Thread Duo Zhang
On behalf of the Apache HBase PMC I am pleased to announce that Bryan Beaudreault has accepted our invitation to become a PMC member on the Apache HBase project. We appreciate Bryan Beaudreault stepping up to take more responsibility in the HBase project. Please join me in welcoming Bryan

Re: Hbase Download Question

2023-10-07 Thread Duo Zhang
Is this problem reproducible? I think lots of developers and users in the community use M1 mac now... Harry Jamison 于2023年10月7日周六 10:30写道: > > I figured out what the problem was.I was untaring this on a mac, and it was > somehow corrupting the file.It was fixed when I did it on a linux machine

[ANNOUNCE] New HBase committer Hui Ruan(阮辉)

2023-09-15 Thread Duo Zhang
On behalf of the Apache HBase PMC, I am pleased to announce that Hui Ruan(frostruan) has accepted the PMC's invitation to become a committer on the project. We appreciate all of Hui's generous contributions thus far and look forward to his continued involvement. Congratulations and welcome, Hui

[ANNOUNCE] New HBase committer Hui Ruan(阮辉)

2023-09-15 Thread Duo Zhang
On behalf of the Apache HBase PMC, I am pleased to announce that Hui Ruan(frostruan) has accepted the PMC's invitation to become a committer on the project. We appreciate all of Hui's generous contributions thus far and look forward to his continued involvement. Congratulations and welcome, Hui

Re: Replication Lag Issue in HBase DR Cluster after Upgrade

2023-08-22 Thread Duo Zhang
ferent, and since all hbase >> 1.x version have been EOL for quite some time, I'm not sure what is >> the easier way to fix the problem, maybe you need to read the code a >> bit more carefully to see how to add the above check in 1.x code line. >> >> Thanks. >

Re: Higher availability for writes

2023-08-22 Thread Duo Zhang
AFAIK, currently we can not promote a secondary replica to a primary replica, they are assigned separately. Promoting secondary replica to primary replica has been discussed in the past but hasn't been implemented yet. Thanks. Michiel de Jong 于2023年8月22日周二 18:25写道: > > Hello, > > We are

Re: Replication Lag Issue in HBase DR Cluster after Upgrade

2023-08-20 Thread Duo Zhang
1.x version have been EOL for quite some time, I'm not sure what is the easier way to fix the problem, maybe you need to read the code a bit more carefully to see how to add the above check in 1.x code line. Thanks. Valli 于2023年8月17日周四 23:14写道: > > Hi Duo Zhang > > Its just m

Re: Replication Lag Issue in HBase DR Cluster after Upgrade

2023-08-15 Thread Duo Zhang
Is this just a metrics issue or is there an actual replication lag? Valli 于2023年8月11日周五 22:51写道: > > Hello HBase Community, > > We recently upgraded our HBase cluster from version 1.2.6 to 1.4.14 and > have encountered an issue with replication lag in our Disaster Recovery > (DR) cluster. We

Re: [DISCUSS] Reschedule July meetup?

2023-07-25 Thread Duo Zhang
We'd better discuss the topics we want to discuss in the online meeting on the mailing list first, and post them out when scheduling the meeting. In this way I think we can attract more attendees. And for me, I prefer a night session, as I have to take care of my children in the morning...

Re: [DISCUSS] Reschedule July meetup?

2023-07-25 Thread Duo Zhang
We'd better discuss the topics we want to discuss in the online meeting on the mailing list first, and post them out when scheduling the meeting. In this way I think we can attract more attendees. And for me, I prefer a night session, as I have to take care of my children in the morning...

[ANNOUNCE] Apache HBase 3.0.0-alpha-4 is now available for download

2023-06-08 Thread Duo Zhang
The HBase team is happy to announce the immediate availability of HBase 3.0.0-alpha-4. Apache HBase™ is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To

[ANNOUNCE] Apache HBase 3.0.0-alpha-4 is now available for download

2023-06-08 Thread Duo Zhang
The HBase team is happy to announce the immediate availability of HBase 3.0.0-alpha-4. Apache HBase™ is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To

Re: Hmaster starting issue.

2023-06-07 Thread Duo Zhang
Do you want to start a single node pseudo distributed hbase cluster, or a fully distributed cluster? For the former one, you do not need to start hdfs or zookeeper, just type ./start-hbase.sh with default configuration is enough. If you want a fully distributed cluster, and want it to use pre

Re: Call for Presentations, Community Over Code Asia 2023

2023-06-05 Thread Duo Zhang
native, no other self deployed services other than HBase itself - New WAL implementation: BookKeeper? Embedded WAL service? Plan to create a google doc online so others in the community could also help polishing the slides, as I do not know all the details about these features. Thanks. Duo

Re: Call for Presentations, Community Over Code Asia 2023

2023-06-05 Thread Duo Zhang
native, no other self deployed services other than HBase itself - New WAL implementation: BookKeeper? Embedded WAL service? Plan to create a google doc online so others in the community could also help polishing the slides, as I do not know all the details about these features. Thanks. Duo

Fwd: Call for Presentations, Community Over Code Asia 2023

2023-06-05 Thread Duo Zhang
FYI. The CFP deadline has been extended to 6.18, please submit your representations :) 会议的中文介绍见:https://www.bagevent.com/event/8409854 -- Forwarded message - 发件人: Rich Bowen Date: 2023年6月6日周二 00:09 Subject: Call for Presentations, Community Over Code Asia 2023 To:

Fwd: Call for Presentations, Community Over Code Asia 2023

2023-06-05 Thread Duo Zhang
FYI. The CFP deadline has been extended to 6.18, please submit your representations :) 会议的中文介绍见:https://www.bagevent.com/event/8409854 -- Forwarded message - 发件人: Rich Bowen Date: 2023年6月6日周二 00:09 Subject: Call for Presentations, Community Over Code Asia 2023 To:

Re: HBase2 Orphan Regions on RegionServer

2023-06-05 Thread Duo Zhang
建议 grep 一下 master 的日志,这种大概率是已经被合并的 region,只是没有正确的 offline,可以看看当前表里和他重叠的那个 region 是怎么合并出来的 在 2.3 以后这种情况会少很多,主要是 AM-v2 和 procedure 存储的部分做了不少改进和优化 leojie 于2023年6月5日周一 16:09写道: > hi all > 请教社区一些HBase2中 Orphan Regions的问题,使用的hbase版本是2.2.6,hadoop版本是3.3.2 > HBase的hbck report页面出现了一些Orphan Regions, >

Re: TimeoutIOException: Failed to get sync result after 300000 ms for txid=16920651960, WAL system stuck?

2023-05-31 Thread Duo Zhang
, > > 停止该任务后,这两天集群指标比较稳定了,没有发现WAL system stuck?的异常了。 > > > > 感觉DN有压力是引起此现象的一个导火索,但可能不是sync > > wal操作hang住一两个小时后,才导致RS挂掉的直接原因,我试试升级jdk,再研读下这块的代码 > > > > 张铎(Duo Zhang) 于2023年5月13日周六 18:58写道: > > > >> 这看起来还是网络有抖动所以连接断了?建议还是查查出这个 log 时候集群的各种指标 > >

Re: TimeoutIOException: Failed to get sync result after 300000 ms for txid=16920651960, WAL system stuck?

2023-05-13 Thread Duo Zhang
iled > org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException: > syscall:read(..) failed: Connection reset by peer > at > > org.apache.hbase.thirdparty.io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown > Source) > 2023-05-11 10:59:33,229 INFO [HBase

Re: TimeoutIOException: Failed to get sync result after 300000 ms for txid=16920651960, WAL system stuck?

2023-05-10 Thread Duo Zhang
你往上翻翻有没有别的异常?这个看起来应该就是 AsyncFSWAL 有 bug 导致 hang 住不动了,不过我翻了一下,2.2.6 之后似乎没有跟这个有关的 fix 了,之前倒是有一些。 另外你的 jdk 版本是多少?我印象里 jdk8 早期版本 synchronized 有个 bug 可能会导致执行顺序错乱 leojie 于2023年5月9日周二 14:45写道: > hi all > 向社区求助一个问题,这两天总是在12:50左右遇到一个异常,描述如下: > hbase版本:2.2.6 > hadoop版本:3.3.1 >

Re: 关于offPeakCompactionTracker的疑惑

2023-05-04 Thread Duo Zhang
可以的,建 issue 吧,先不着急改具体代码 或者你可以内部先改改测试一下,然后把结果啥的也贴到 issue 上 章啸 于2023年5月4日周四 13:35写道: > 我是发现OffPeak compaction并不能充分利用低峰期的集群资源去合并高峰期生成的hfile,因为同一时间只能有一个 off peak > compact。 > 似乎这样并不是很合理,我想尝试修改这里(去掉static,改成一个store > 内同一时间只有一个),但是不理解这样设计的初衷,以及我这样修改是否有问题。 > 我想我可以创建一个jira,把我的修改思路提上来。 > > > 在

Re: 关于offPeakCompactionTracker的疑惑

2023-05-03 Thread Duo Zhang
我 blame 翻了一下 https://github.com/apache/hbase/blob/5998a0f349824adf823f79a52530e97dfc624b92/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/OffPeakCompactions.java 这个 AtomicBoolean 的作用其实就是替代这个文件里的一些逻辑的,在 HBASE-7437 的改动里把这个文件去掉了

Re: [ANNOUNCE] New HBase committer Nihal Jain

2023-05-03 Thread Duo Zhang
Congratulations! Viraj Jasani 于2023年5月3日周三 23:47写道: > Congratulations Nihal!! Very well deserved!! > > On Wed, May 3, 2023 at 5:12 AM Nick Dimiduk wrote: > > > Hello! > > > > On behalf of the Apache HBase PMC, I am pleased to announce that Nihal > Jain > > has accepted the PMC's invitation to

Re: [ANNOUNCE] New HBase committer Nihal Jain

2023-05-03 Thread Duo Zhang
Congratulations! Viraj Jasani 于2023年5月3日周三 23:47写道: > Congratulations Nihal!! Very well deserved!! > > On Wed, May 3, 2023 at 5:12 AM Nick Dimiduk wrote: > > > Hello! > > > > On behalf of the Apache HBase PMC, I am pleased to announce that Nihal > Jain > > has accepted the PMC's invitation to

Re: Re: Re: hbase集群单个region flush时间过长问题求助

2023-04-02 Thread Duo Zhang
2.1 上的代码我记的不太清楚了,不过 region assign 相关 procedure 在 2.2 重新写了一遍,之前确实有 bug 很难搞定,常见的就是 region 不下线,或者一个 region 被 assign 到两台 RS 上,只能重启搞定。另外后来又在 2.3 上把存储 procedure 的部分重新实现了一遍,才算比较稳定。 所以 2.1 在大集群特别是有些过载的情况下跑 merge 确实有可能出 bug,hbck 的处理没啥问题,也只能这样搞。。。 邢* 于2023年3月31日周五 15:15写道: > >

Re: 大表快照scan场景中,遇到快照删除,但是快照引用的hfile未删除的现象

2023-04-02 Thread Duo Zhang
嗯,逻辑上没问题,就是写法,你可以先在自己的仓库上修了试一下 leojie 于2023年4月3日周一 09:04写道: > 感谢张老师的回复,这个PR是我们组之前一个同事提的,,这个修复没啥问题,应该是可以直接应用的吧 > > 张铎(Duo Zhang) 于2023年3月30日周四 12:09写道: > > > https://issues.apache.org/jira/browse/HBASE-25510 > > > > 这个,不过还没合,也很久没动静了,我再跟一下吧 > > > > leoj

Re: Re: hbase集群单个region flush时间过长问题求助

2023-03-30 Thread Duo Zhang
> is closed by DFSClient_NONMAPREDUCE_-1935786627_1 > 2023-03-30 12:22:53,311 DEBUG org.apache.hadoop.net.NetworkTopology: > Choosing random from 13 available nodes on node /, scope=, > excludedScope=/TX/T01/03, excludeNodes=[x.x.x.x:9866]. numOfDatanodes=13. > 2023-03-30 12:22:53,311 DEBUG org.apache.h

Re: 大表快照scan场景中,遇到快照删除,但是快照引用的hfile未删除的现象

2023-03-29 Thread Duo Zhang
https://issues.apache.org/jira/browse/HBASE-25510 这个,不过还没合,也很久没动静了,我再跟一下吧 leojie 于2023年3月30日周四 11:43写道: > 好的,感谢张老师回复,我搜下ISSUE列表 > > 张铎(Duo Zhang) 于2023年3月29日周三 21:44写道: > > > 这都是 cleaner 的线程,能跑到这里说明还是在扫描看文件是否可以被删除的,并不是判断有 snapshot 正在执行就直接跳过执行? > > > > 可以看看是不是就是删

Re: hbase集群单个region flush时间过长问题求助

2023-03-29 Thread Duo Zhang
这个要看你 flush 出去的数据有多大吧?以及写 HDFS 的速度,十几秒通常看还行吧,不算特别慢。有监控打 snapshot 时候集群的整体负载和 IO 情况吗? 邢* 于2023年3月29日周三 19:20写道: > hi all, > > > 向社区求助一个HBase的情况,情况描述如下:我们hbase集群某张大表最近做snapshot的时间过长,耗时约半小时。梳理日志发现每个region > flush的时间在十几秒左右,想咨询下flush的时间为什么会这么久? > > 集群现状:目前是有14台cpu内存都是96c

Re: 大表快照scan场景中,遇到快照删除,但是快照引用的hfile未删除的现象

2023-03-29 Thread Duo Zhang
ter.cleaner.CleanerChore.deleteAction(CleanerChore.java:442) > at > org.apache.hadoop.hbase.master.cleaner.CleanerChore.traverseAndDelete(CleanerChore.java:387) > at > org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$null$2(CleanerChore.java:396) > at > org.apache.had

Re: 大表快照scan场景中,遇到快照删除,但是快照引用的hfile未删除的现象

2023-03-27 Thread Duo Zhang
那个日志的意思是有 snapshot 正在执行,所以不会跑 cleaner。可以查一下是否有正在执行的 snapshot 操作卡住了之类的 leojie 于2023年3月27日周一 20:53写道: > hi all, > > 向社区求助一个HBase的情况,情况描述如下:在我们快照scan的场景中,有些大表会较为频繁的做快照(如天级),但是这些快照元数据删除后,快照引用的hfile貌似未清理,这体现在,我们的集群archive目录空间占用嗖嗖往上涨。hmaster中,只找到如下貌似相关的日志: > [image: image.png] > 2023-03-27

[ANNOUNCE] New HBase committer Tianhang Tang(唐天航)

2023-03-15 Thread Duo Zhang
On behalf of the Apache HBase PMC, I am pleased to announce that Tianhang Tang(thangTang) has accepted the PMC's invitation to become a committer on the project. We appreciate all of Tianhang's generous contributions thus far and look forward to his continued involvement. Congratulations and

[ANNOUNCE] New HBase committer Tianhang Tang(唐天航)

2023-03-15 Thread Duo Zhang
On behalf of the Apache HBase PMC, I am pleased to announce that Tianhang Tang(thangTang) has accepted the PMC's invitation to become a committer on the project. We appreciate all of Tianhang's generous contributions thus far and look forward to his continued involvement. Congratulations and

Re: [DISCUSS] How to deal with the disabling of public sign ups for jira.a.o(enable github issues?)

2023-03-06 Thread Duo Zhang
/apache/hbase/pull/5088 PTAL. Thanks. Guanghao Zhang 于2022年12月7日周三 12:35写道: > Did other projects have the same solution for this, sync github issues to > jira issues? Github issues will be useful to get more feedback. > > 张铎(Duo Zhang) 于2022年12月6日周二 00:13写道: > > > T

Re: ReadOnlyZKClient question

2023-02-08 Thread Duo Zhang
These are all normal logs for ReadOnlyZkClient, but the problem is we should not output these logs when running hbase shell. I'm not a ruby expert but we have this in the entry point of hbase shell(jar-bootstrap.rb) # Set logging level to avoid verboseness

[ANNOUNCE] Apache HBase 2.4.16 is now available for download

2023-02-05 Thread Duo Zhang
The HBase team is happy to announce the immediate availability of HBase 2.4.16. Apache HBase™ is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To learn

[ANNOUNCE] Apache HBase 2.4.16 is now available for download

2023-02-05 Thread Duo Zhang
The HBase team is happy to announce the immediate availability of HBase 2.4.16. Apache HBase™ is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To learn

HBase Quarterly report Oct-Dec 2022

2023-02-05 Thread Duo Zhang
Hi all, HBase submits a report to the ASF board once a quarter, to inform the board about project health. I'm sending the report to the user@ and dev@ mailing lists because you are the project, and for transparency. If you have any questions about the report or the running of the project, you can

HBase Quarterly report Oct-Dec 2022

2023-02-05 Thread Duo Zhang
Hi all, HBase submits a report to the ASF board once a quarter, to inform the board about project health. I'm sending the report to the user@ and dev@ mailing lists because you are the project, and for transparency. If you have any questions about the report or the running of the project, you can

[ANNOUNCE] Please welcome Tak Lon (Stephen) Wu to the HBase PMC

2023-01-29 Thread Duo Zhang
On behalf of the Apache HBase PMC I am pleased to announce that Tak Lon (Stephen) Wu has accepted our invitation to become a PMC member on the Apache HBase project. We appreciate Tak Lon (Stephen) Wu stepping up to take more responsibility in the HBase project. Please join me in welcoming Tak Lon

[ANNOUNCE] Please welcome Tak Lon (Stephen) Wu to the HBase PMC

2023-01-29 Thread Duo Zhang
On behalf of the Apache HBase PMC I am pleased to announce that Tak Lon (Stephen) Wu has accepted our invitation to become a PMC member on the Apache HBase project. We appreciate Tak Lon (Stephen) Wu stepping up to take more responsibility in the HBase project. Please join me in welcoming Tak Lon

Re: scan 引发未捕获运行时异常导致RS宕机

2023-01-27 Thread Duo Zhang
instanceof HFileBlock)) { > comparison = ((HFileBlock) existing).getNextBlockOnDiskSize() > - ((HFileBlock) newBlock).getNextBlockOnDiskSize(); > } > return comparison; > } > > 这里comparison != 0这个判断逻辑是必须的? > > > 张铎(Duo Zhang) 于2023年1月28日周六 11:09写道: > >

Re: scan 引发未捕获运行时异常导致RS宕机

2023-01-27 Thread Duo Zhang
建议先升级到2.4或者2.5的最新版试试?2.2.6 已经是比较老的版本了,后面 BucketCache 修过不少小 bug https://issues.apache.org/jira/browse/HBASE-26281 比如这个就有可能导致BucketCache里的内容错乱,出现各种奇怪的错误 leojie 于2023年1月28日周六 10:22写道: > > Hi all, > 向社区求助一个HBase的scan异常导致RS宕机的问题,异常日志如下: > > 2023-01-19 03:19:06,986 ERROR >

Re: [DISCUSS] Allow namespace admins to clone snapshots created by them

2023-01-18 Thread Duo Zhang
oner release necessary. > > If the change is accepted could I please get a review for the PR? > https://github.com/apache/hbase/pull/4885 > > On Wed, Jan 4, 2023 at 3:17 AM 张铎(Duo Zhang) wrote: > > > +1 on releasing 2.6.0 sooner. > > > > And I think it is time to

Re: [DISCUSS] Allow namespace admins to clone snapshots created by them

2023-01-03 Thread Duo Zhang
> > >> > > >> This should not break any existing use case so I see no reason to not > > add > > >> this to branch-2.5 and > > >> branch-2.4. > > >> > > >>> On Thu, Nov 24, 2022 at 3:03 AM 张铎(Duo Zhang) > > wrote

Re: [DISCUSS] Allow namespace admins to clone snapshots created by them

2022-12-23 Thread Duo Zhang
ak any existing use case so I see no reason to not add > this to branch-2.5 and > branch-2.4. > > On Thu, Nov 24, 2022 at 3:03 AM 张铎(Duo Zhang) wrote: > > > I'm OK with this change. > > > > But maybe we still need to determine which branches we can apply this > &g

[ANNOUNCE] New HBase committer Rushabh Shah

2022-12-14 Thread Duo Zhang
On behalf of the Apache HBase PMC, I am pleased to announce that Rushabh Shah(shahrs87) has accepted the PMC's invitation to become a committer on the project. We appreciate all of Rushabh's generous contributions thus far and look forward to his continued involvement. Congratulations and

[ANNOUNCE] New HBase committer Rushabh Shah

2022-12-14 Thread Duo Zhang
On behalf of the Apache HBase PMC, I am pleased to announce that Rushabh Shah(shahrs87) has accepted the PMC's invitation to become a committer on the project. We appreciate all of Rushabh's generous contributions thus far and look forward to his continued involvement. Congratulations and

Re: [DISCUSS] How to deal with the disabling of public sign ups for jira.a.o(enable github issues?)

2022-12-05 Thread Duo Zhang
The PR for HBASE-27513 is available https://github.com/apache/hbase/pull/4913 Let's at least tell our users to send email to private@hbase for acquiring a jira account. Thanks. 张铎(Duo Zhang) 于2022年12月2日周五 12:46写道: > > Currently all the comment on github PR will be sent to issues@hbase,

[ANNOUNCE] Apache HBase 2.5.2 is now available for download

2022-12-04 Thread Duo Zhang
The HBase team is happy to announce the immediate availability of HBase 2.5.2. Apache HBase™ is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To learn

[ANNOUNCE] Apache HBase 2.5.2 is now available for download

2022-12-04 Thread Duo Zhang
The HBase team is happy to announce the immediate availability of HBase 2.5.2. Apache HBase™ is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To learn

Re: [ANNOUNCE] New HBase Committer Liangjun He

2022-12-03 Thread Duo Zhang
Congratulations! Yu Li 于2022年12月3日周六 21:51写道: > > Hi All, > > On behalf of the Apache HBase PMC, I am pleased to announce that Liangjun > He (heliangjun) has accepted the PMC's invitation to become a committer on > the project. We appreciate all of Liangjun's generous contributions thus > far

Re: [ANNOUNCE] New HBase Committer Liangjun He

2022-12-03 Thread Duo Zhang
Congratulations! Yu Li 于2022年12月3日周六 21:51写道: > > Hi All, > > On behalf of the Apache HBase PMC, I am pleased to announce that Liangjun > He (heliangjun) has accepted the PMC's invitation to become a committer on > the project. We appreciate all of Liangjun's generous contributions thus > far

Re: [DISCUSS] How to deal with the disabling of public sign ups for jira.a.o(enable github issues?)

2022-12-01 Thread Duo Zhang
l, but maybe an option? I'm guessing it would require INFRA help. > > On Thu, Dec 1, 2022 at 5:47 AM 张铎(Duo Zhang) wrote: > > > I've filed HBASE-27513 for changing the readme on github. > > > > At least let's reuse the existing mailing list for acquiring jira account. > &

Re: [DISCUSS] How to deal with the disabling of public sign ups for jira.a.o(enable github issues?)

2022-12-01 Thread Duo Zhang
I've filed HBASE-27513 for changing the readme on github. At least let's reuse the existing mailing list for acquiring jira account. Thanks. 张铎(Duo Zhang) 于2022年11月29日周二 22:34写道: > > Bump and also send this to user@hbase. > > We need to find a way to deal with the current sit

Re: [DISCUSS] How to deal with the disabling of public sign ups for jira.a.o(enable github issues?)

2022-11-29 Thread Duo Zhang
... Thanks. 张铎(Duo Zhang) 于2022年11月27日周日 22:06写道: > > For me, I think most developers already have a github account, so > enabling it could help us get more feedback. For lots of younger > Chinese developers, they rarely use email in their daily life... > No doubt later we need to mo

Re: [DISCUSS] Allow namespace admins to clone snapshots created by them

2022-11-23 Thread Duo Zhang
I'm OK with this change. But maybe we still need to determine which branches we can apply this change to? Is it OK to include this change for branch-2.5 and branch-2.4? Tak Lon (Stephen) Wu 于2022年11月22日周二 06:31写道: > > FYI the PR is https://github.com/apache/hbase/pull/4885 and >

Re: [DISCUSS] Quickly release 2.5.2 with hadoop3 binaries

2022-11-21 Thread Duo Zhang
There is still a critical problem, filed HBASE-27501. 张铎(Duo Zhang) 于2022年11月20日周日 23:00写道: > > After fixing the jackson CVEs, I think it is time to make a new release now. > > Will put up 2.5.2RC0 next week. > > Thanks. > > 张铎(Duo Zhang) 于2022年11月4日周五 23:24写道: >

Re: [DISCUSS] Quickly release 2.5.2 with hadoop3 binaries

2022-11-20 Thread Duo Zhang
After fixing the jackson CVEs, I think it is time to make a new release now. Will put up 2.5.2RC0 next week. Thanks. 张铎(Duo Zhang) 于2022年11月4日周五 23:24写道: > > If there are no big concerns, I will put up a 2.5.2RC0 next week, with > hadoop3 binaries, to see if it could meet

Re: [ANNOUNCE] Changes to Jira Account Creation (issues.a.o/jira)

2022-11-19 Thread Duo Zhang
The hive project decided to use a dedicated mailing list for requiring a jira account. https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-JIRA And maybe we could enable github issues to let users ask questions and report bugs there first, if we decided it is a true

Re: [DISCUSS] Quickly release 2.5.2 with hadoop3 binaries

2022-11-04 Thread Duo Zhang
If there are no big concerns, I will put up a 2.5.2RC0 next week, with hadoop3 binaries, to see if it could meet our users' requirements. Thanks. 张铎(Duo Zhang) 于2022年10月31日周一 14:38写道: > > emmm, after reading the title, seems the word 'quickly' is a bit confusing > > What I mean he

Re: [DISCUSS] Quickly release 2.5.2 with hadoop3 binaries

2022-10-31 Thread Duo Zhang
than before, as we want to change the content of the release. Thanks. 张铎(Duo Zhang) 于2022年10月31日周一 11:58写道: > > Recently I've been working on HBASE-27359 together with some other > folks in the community(Nick, Andrew, Guanghao, etc.). After landing > some important improvements, s

[DISCUSS] Quickly release 2.5.2 with hadoop3 binaries

2022-10-30 Thread Duo Zhang
Recently I've been working on HBASE-27359 together with some other folks in the community(Nick, Andrew, Guanghao, etc.). After landing some important improvements, such as HBASE-27434, HBASE-27442 and more, now we are ready to make our release scripts able to publish binaries for hadoop3.

HBase Quarterly report Jul-Sep 2022

2022-10-22 Thread Duo Zhang
Hi all, HBase submits a report to the ASF board once a quarter, to inform the board about project health. I'm sending the report to the user@ and dev@ mailing lists because you are the project, and for transparency. If you have any questions about the report or the running of the project, you can

  1   2   3   4   >