[jira] [Resolved] (HDFS-17626) Reduce lock contention at datanode startup

2024-09-28 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17626. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Reduce lock content

[jira] [Resolved] (HDFS-17624) Fix DFSNetworkTopology#chooseRandomWithStorageType() availableCount when excluded node is not in selected scope.

2024-09-27 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17624. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix DFSNetworkTopol

[INFO] branch HDFS-17531 rebase onto trunk

2024-09-12 Thread Xiaoqiao He
Hi All, Just rebase development feature branch HDFS-17531(RBF: Asynchronous router RPC) onto trunk, then keep branch HDFS-17531 closely with trunk. Thanks. Best Regards, - He Xiaoqiao

[jira] [Resolved] (HDFS-17573) Allow turn on both FSImage parallelization and compression

2024-08-25 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17573. Fix Version/s: (was: 3.4.1) Hadoop Flags: Reviewed Target Version/s: (was

[jira] [Resolved] (HDFS-16084) getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv() failed

2024-08-25 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-16084. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > getJNIEnv() retu

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

2024-08-22 Thread Xiaoqiao He
+1 (binding). Thanks Steve and Mukund for driving this release. Best Regards - He Xiaoqiao On Wed, Aug 21, 2024 at 11:51 PM Mukund Madhav Thakur wrote: > +1 ( binding) > > On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran > > wrote: > > > Mukund and I have built a release candidate (RC1) for Had

[jira] [Resolved] (HDFS-17544) [ARR] The router client rpc protocol PB supports asynchrony.

2024-08-02 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17544. Fix Version/s: HDFS-17531 Hadoop Flags: Reviewed Resolution: Fixed > [ARR] The rou

[jira] [Resolved] (HDFS-16690) Automatically format new unformatted JournalNodes using JournalNodeSyncer

2024-07-23 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-16690. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Automatically for

[jira] [Resolved] (HDFS-17566) Got wrong sorted block order when StorageType is considered.

2024-07-11 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17566. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Got wrong sorted bl

[jira] [Resolved] (HDFS-17564) EC: Fix the issue of inaccurate metrics when decommission mark busy DN

2024-07-05 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17564. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > EC: Fix the issue

Re: [DISCUSS] lots of yetus failures in hdfs

2024-06-07 Thread Xiaoqiao He
mzl61ow0sbs10p0hky17xxhsggbhc3g > [4] https://github.com/apache/hadoop/pull/6664#issuecomment-2082356393 > > On Fri, 7 Jun 2024 at 08:04, Xiaoqiao He wrote: > > > > Thanks Steve. Try to trigger CI manually and let's wait what it will say. > > BTW, the flaky tests seem not r

Re: [DISCUSS] lots of yetus failures in hdfs

2024-06-06 Thread Xiaoqiao He
Thanks Steve. Try to trigger CI manually and let's wait what it will say. BTW, the flaky tests seem not related to UT logic itself, but most of them throw OOM. Not sure if @Ayush Saxena knows how to re-config or tune the memory of Yetus? Best Regards, - He Xiaoqiao On Fri, Jun 7, 2024 at 3:59 AM

Re: [Discuss] RBF: Aynchronous router RPC.

2024-06-05 Thread Xiaoqiao He
Hi @zhangjian <1361320...@qq.com> , the dev branch HDFS-17531 is ready now. FYI. Best Regards, - He Xiaoqiao On Tue, Jun 4, 2024 at 10:08 PM zhangjian <1361320...@qq.com> wrote: > Hi, Xiaoqiao He: > Can you help create a dev branch? I don't have the permission to cre

Re: [Discuss] RBF: Aynchronous router RPC.

2024-05-29 Thread Xiaoqiao He
> On Fri, May 24, 2024 at 1:57 AM zhangjian <1361320...@qq.com> wrote: > >>>>> Hello everyone, currently, I have tested the performance of async > and sync router for a downstream ns: > >>>>> 1. The throughput, CPU, and thread performance of the a

Re: [DISCUSS] pruning surplus branches/tags from the main hadoop repo

2024-05-29 Thread Xiaoqiao He
Strong +1. One concerns, how we define 'unnecessary branches', which mean how to distinguish the branches someone accidentally created, I try to traverse some of them but didn't get one obvious rule. Thanks. Best Regards, - He Xiaoqiao On Wed, May 29, 2024 at 10:11 PM Ayush Saxena wrote: > +1

Re: [Discuss] RBF: Aynchronous router RPC.

2024-05-23 Thread Xiaoqiao He
gjin > > > > On Tue, May 21, 2024 at 5:13 AM zhangjian <1361320...@qq.com.invalid> > wrote: > > > >> Hi, xiaoqiao he, thank you for your reply. > >> > >> 1.Currently, the server and client protocols within router can be >

Re: [Discuss] RBF: Aynchronous router RPC.

2024-05-20 Thread Xiaoqiao He
Thanks for this great proposal! Some questions after reviewing the design doc (sorry didn't review PR carefully which is too large.) 1. This solution will involve RPC framework update, will it affect other modules and how to keep other modules off these changes. 2. Some RPC requests should be forw

[jira] [Resolved] (HDFS-17099) Fix Null Pointer Exception when stop namesystem in HDFS

2024-05-13 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17099. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix Null Poin

[jira] [Resolved] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection

2024-05-13 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17522. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > JournalNode

[jira] [Resolved] (HDFS-17500) Add missing operation name while authorizing some operations

2024-05-05 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17500. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Add missing operat

[jira] [Resolved] (HDFS-17456) Fix the incorrect dfsused statistics of datanode when appending a file.

2024-04-29 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17456. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix the incorr

Re: Discussion about NameNode Fine-grained locking

2024-04-28 Thread Xiaoqiao He
ts: >> > >> > 1. While ZanderXu and Hui Fei have deep expertise in HDFS and are familiar >> > with related development details, we still need more community member to >> > review the code to ensure that the relevant upgrades meet expectations. >> > >>

Re: Discussion about NameNode Fine-grained locking

2024-04-24 Thread Xiaoqiao He
cc private@h.a.o. On Wed, Apr 24, 2024 at 3:35 PM ZanderXu wrote: > > Here are some summaries about the first phase: > 1. There are no big changes in this phase > 2. This phase just uses FS lock and BM lock to replace the original global > lock > 3. It's useful to improve the performance, since s

[ANNOUNCE] New Hadoop Committer - Haiyang Hu

2024-04-21 Thread Xiaoqiao He
I am pleased to announce that Haiyang Hu has been elected as a committer on the Apache Hadoop project. We appreciate all of Haiyang's work, and look forward to her/his continued contributions. Congratulations and Welcome, Haiyang! Best Regards, - He Xiaoqiao (On behalf of the Apache Hadoop PMC)

[jira] [Resolved] (HDFS-17461) Fix spotbugs in PeerCache#getInternal

2024-04-12 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17461. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix spotbugs

[jira] [Resolved] (HDFS-17455) Fix Client throw IndexOutOfBoundsException in DFSInputStream#fetchBlockAt

2024-04-11 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17455. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix Client th

[jira] [Resolved] (HDFS-17368) HA: Standy should exit safemode when resources are from low available

2024-03-26 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17368. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > HA: Standy should e

[jira] [Resolved] (HDFS-17380) FsImageValidation: remove inaccessible nodes

2024-03-17 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17380. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > FsImageValidat

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

2024-03-13 Thread Xiaoqiao He
root root 2739952 Sep 19 13:09 >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1 >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40 >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3 >> >> Anyone got any insights on how I should set up this (debian-based) OS >> here? >>

[jira] [Resolved] (HDFS-17391) Adjust the checkpoint io buffer size to the chunk size

2024-03-12 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17391. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Adjust the checkpo

[jira] [Resolved] (HDFS-17422) Enhance the stability of the unit test TestDFSAdmin

2024-03-11 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17422. Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Enhance the stabil

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

2024-03-11 Thread Xiaoqiao He
x27;ve > > verified they are good. > > > > Oh, and I've also got my raspberry p5 doing the download of the arm > > stuff for its checknative; not expecting problems. > > > > So: i've got some stuff still ongoing, but the core changes to packaging > &g

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

2024-03-11 Thread Xiaoqiao He
fixed in HADOOP-19057 > <https://issues.apache.org/jira/browse/HADOOP-19057>. > > -- > *From:* slfan1989 > *Sent:* Monday, March 4, 2024 10:07:05 PM > *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; < > priv...@hadoop.apache.org>

Re: Discussion about NameNode Fine-grained locking

2024-03-06 Thread Xiaoqiao He
ful. > > > > > > > > > > > > > > > > On Wed, Mar 6, 2024 at 10:51 AM Hui Fei > wrote: > > > > > > > > > Thanks for suggestions. > > > > > > > > > > Actually Started working on this improvement. And c

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

2024-03-06 Thread Xiaoqiao He
cc @Chao Sun @zhang...@apache.org @Ayush Saxena Would you mind helping to check the upstream systems(Spark, HBase, Hive) dependency with the new hadoop release version if necessary? Thanks. Best Regards, - He Xiaoqiao On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He wrote: > Shilun, Thanks

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

2024-03-06 Thread Xiaoqiao He
Shilun, Thanks for your great work. I am checking but not finished yet. Will confirm once done. Best Regards, - He Xiaoqiao On Thu, Mar 7, 2024 at 7:23 AM slfan1989 wrote: > @Xiaoqiao He @Ayush Saxena > @Steve > Loughran @Mukund Madhav Thakur > @Takanobu > Asanuma @M

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

2024-02-29 Thread Xiaoqiao He
files and run some test with it; this caught some >>> problems >>> > >> with exported slf4j and log4j2 artifacts getting into the >>> classpath. That >>> > >> is: hadoop-common pulling in log4j 1.2 and 2.x bindings. >>> > >> &g

Re: Discussion about NameNode Fine-grained locking

2024-02-26 Thread Xiaoqiao He
Thanks for this meaningful proposal. Some nit comments: a. Snapshot, Symbolic link and reserved feature are not mentioned at the design doc, should it be considered or different to this core design? b. For the benchmark result, what Read/Write request ratio? And do you meet any GC issues when reach

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 (RC1)

2024-02-05 Thread Xiaoqiao He
cc @PJ Fanning @Ayush Saxena @Steve Loughran @Takanobu Asanuma @Shuyan Zhang @inigo...@apache.org On Mon, Feb 5, 2024 at 11:30 AM Xiaoqiao He wrote: > +1(binding). > > I checked the following items: > - [X] Download links are valid. > - [X] Checksums and PGP signatures ar

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 (RC1)

2024-02-04 Thread Xiaoqiao He
10:29 PM slfan1989 wrote: > Hi folks, > > Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop > Thirdparty 1.2.0. > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC1 > > The RC tag is > ht

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

2024-02-01 Thread Xiaoqiao He
ds, > Shilun Fan. > > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He wrote: > > > Thanks Shilun for driving it and making it happen. > > > > +1(binding). > > > > [x] Checksums and PGP signatures are valid. > > [x] LICENSE files exist. > > [x] NOTICE

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

2024-01-30 Thread Xiaoqiao He
wrote: > Hi folks, > > Xiaoqiao He and I have put together a release candidate (RC0) for Hadoop > Thirdparty 1.2.0. > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0 > > The RC tag is > https://github.com/apache/

Re: [INFO] Hadoop-3.4 Release Update

2024-01-24 Thread Xiaoqiao He
Mon, 22 Jan 2024 at 11:50 AM, slfan1989 wrote: > > > >> Thank you very much for specifying the frozen time for the > >> Branch-3.4.0. @Xiaoqiao > >> He > >> > >> If there is a need to backport any PRs to branch-3.4/branch-3.4.0, > please &

Re: [INFO] Hadoop-3.4 Release Update

2024-01-21 Thread Xiaoqiao He
gt; > Best > > On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He > wrote: > >> Thanks all for your input. Connected to Shilun offline yesterday, and he >> will >> update or recover the fix version tag before 3.4.0-RC1. >> >> Best Regards, >> - He Xiao

Re: [INFO] Hadoop-3.4 Release Update

2024-01-21 Thread Xiaoqiao He
se versions have been released, > remove > > > the 3.4.0 entry from the fix version. > > > > > > 3. For JIRA issues with target version set to 3.4.0, fix version set to > > > 3.4.0, and status set to > > > RESOLVED, if Affects Ve

[INFO] Hadoop-3.4 Release Update

2024-01-18 Thread Xiaoqiao He
Hi All, Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is created[1]. Please set the proper fix version while committing jira, 3.5.0 should be priority. If there is any blocker/critical please also backport to branch-3.4 or sync to me or RM Shilun Fan. Shilun will start 3.4.0 RC1 vot

[jira] [Resolved] (HDFS-17182) DataSetLockManager.lockLeakCheck() is not thread-safe.

2024-01-03 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17182. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed

Re: [DISCUSS] Release Hadoop 3.4.0

2024-01-03 Thread Xiaoqiao He
> > I think the release discussion can be in public ML? Good idea. cc common-dev/hdfs-dev/yarn-dev/mapreduce-dev ML. Best Regards, - He Xiaoqiao On Tue, Jan 2, 2024 at 6:18 AM Ayush Saxena wrote: > +1 from me as well. > > We should definitely attempt to upgrade the thirdparty version for > 3.

Re: [VOTE] Hadoop 3.2.x EOL

2023-12-21 Thread Xiaoqiao He
Hi All, JIRA/WIKI/Website has been updated. Release line 3.2 is EOL now. Thanks all. Best Regards, - He Xiaoqiao On Tue, Dec 19, 2023 at 11:58 AM Xiaoqiao He wrote: > This vote passed with 10 +1. I'll update the JIRA, WIKI and website for a > while. > > Thanks all for y

Re: [DISCUSS][HDFS] Add rust binding for libhdfs

2023-12-20 Thread Xiaoqiao He
Thanks Xuanwo for your work. I believe it is valuable to enlarge hadoop ecosystem. I am also concerned that it will involve more hard work to release and version match, especially for one who is not familiar with C or Rust. Moreover, I am not aware the difference between `accept hdfs-sys as part o

Re: [VOTE] Hadoop 3.2.x EOL

2023-12-18 Thread Xiaoqiao He
; >> On Wed, Dec 6, 2023 at 8:39 AM Akira Ajisaka wrote: >> > >> > +1 >> > >> > >> > >> > On Wed, Dec 6, 2023 at 1:10 PM Xiaoqiao He wrote: >> > >> > > Dear Hadoop devs, >> > > >> > > Given the

[jira] [Resolved] (HDFS-17262) Transfer rate metric warning log is too verbose

2023-12-06 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17262. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Transfer rate met

[jira] [Resolved] (HDFS-17270) RBF: Fix ZKDelegationTokenSecretManagerImpl use closed zookeeper client to get token in some case

2023-12-06 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17270. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF:

[VOTE] Hadoop 3.2.x EOL

2023-12-05 Thread Xiaoqiao He
Dear Hadoop devs, Given the feedback from the discussion thread [1], I'd like to start an official thread for the community to vote on release line 3.2 EOL. It will include, a. An official announcement informs no further regular Hadoop 3.2.x releases. b. Issues which target 3.2.5 will not be fixe

Re: [DISCUSS] Make some release lines EOL

2023-12-05 Thread Xiaoqiao He
ease, as there's a clear story: we have a > > new 3.4.x line stabilising, if you want something already stable move > onto > > 3.3.x if you hadn't already > > > > > > > > > > > > On Mon, 4 Dec 2023 at 12:39, Xiaoqiao He wrote: > > &g

Re: [DISCUSS] Make some release lines EOL

2023-12-04 Thread Xiaoqiao He
> > -Ayush > > On Mon, 4 Dec 2023 at 18:09, Xiaoqiao He wrote: > > > > Hi folks, > > > > There are many discussions about which release lines should we still > > consider actively > > maintained in history. I want to launch this topic again, and

[jira] [Resolved] (HDFS-17250) EditLogTailer#triggerActiveLogRoll should handle thread Interrupted

2023-12-04 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17250. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > EditLogTai

[jira] [Resolved] (HDFS-17218) NameNode should process time out excess redundancy blocks

2023-12-04 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17218. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > NameNode sho

[DISCUSS] Make some release lines EOL

2023-12-04 Thread Xiaoqiao He
Hi folks, There are many discussions about which release lines should we still consider actively maintained in history. I want to launch this topic again, and try to get a consensus. >From download page[1] and active branches page[2], we have the following release lines: Hadoop 3.3 Release (relea

Re: [DISCUSS] Hadoop 3.4.0 release

2023-12-03 Thread Xiaoqiao He
Thanks Ahmar for starting this thread. Strong +1 to have 3.4.0 from trunk rather than from 3.3.*. We should keep release line clear, and it's not standard operation to cut a branch from another line. Let's move the trunk to 3.5.0-SNAPSHOT when we come to an agreement at this thread. About this rel

[jira] [Resolved] (HDFS-17063) Support to configure different capacity reserved for each disk of DataNode.

2023-11-16 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17063. Fix Version/s: 3.4.0 Resolution: Fixed > Support to configure different capacity reser

[jira] [Resolved] (HDFS-17172) Support FSNamesystemLock Parameters reconfigurable

2023-11-16 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17172. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Supp

Re: Review request for https://issues.apache.org/jira/browse/HDFS-15413

2023-11-02 Thread Xiaoqiao He
Hi Bryan, Thanks for reaching out and sorry for the late response for this JIRA issue. I tried to contact one HDFS EC expert to get involved. Hope we can push this forward. Thanks again. Best Regards, - He Xiaoqiao On Fri, Nov 3, 2023 at 12:49 AM Bryan Beaudreault wrote: > Hello devs, > > Ther

[ANNOUNCE] New Hadoop PMC Member - Shilun Fan

2023-10-31 Thread Xiaoqiao He
On behalf of the Apache Hadoop PMC, I am pleased to announce that Shilun Fan(slfan1989) has accepted the PMC's invitation to become PMC member on the project. We appreciate all of Shilun's generous contributions thus far and look forward to his continued involvement. Congratulations and welcome, S

Re: 关于hadoop-3.3.1版本使用libhdfs3.so 访问hdfs联邦模式router节点rpc端口有20分钟延时问题

2023-10-30 Thread Xiaoqiao He
Add hdfs-dev@h.a.o and user@h.a.o On Thu, Oct 26, 2023 at 7:07 PM 王继泽 wrote: > 最近在使用hadoop的过程中,发现了一个情况。 > 当我使用c > api向hdfs联邦模式router节点rpc端口发送请求时,比如说写文件,客户端发送完成请求后,hadoop端需要20分钟延时文件才有字节大小,延时期间不能对文件进行操作。 > > 客户端这边运行结束之后,hadoop端日志大致过程: > 1.namenode接收到客户端的请求,FSEditLog打印日志。 > 2.blockmanager.BlockPlac

[jira] [Resolved] (HDFS-17231) HA: Safemode should exit when resources are from low to available

2023-10-24 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17231. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > HA: Safemode sho

Re: MODERATE for hdfs-iss...@hadoop.apache.org

2023-10-17 Thread Xiaoqiao He
Hi Sergey Onuchin, Sorry to hear that. But we could not give some suggestions based on the only information you mentioned. If any more on-site information may be better to trace, such as depoy architecture, NameNode log and jstack etc. Based on my practice, I did not receive some cases which delet

[jira] [Resolved] (HDFS-17217) Add lifeline RPC start up log when NameNode#startCommonServices

2023-10-12 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17217. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Marked Resolution/Fix

[jira] [Resolved] (HDFS-17208) Add the metrics PendingAsyncDiskOperations in datanode

2023-10-12 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17208. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add the metr

[ANNOUNCE] New Hadoop Committer - Shuyan Zhang

2023-09-26 Thread Xiaoqiao He
I am pleased to announce that Shuyan Zhang has been elected as a committer on the Apache Hadoop project. We appreciate all of Shuyan's work, and look forward to her/his continued contributions. Congratulations and Welcome, Shuyan! Best Regards, - He Xiaoqiao (On behalf of the Apache Hadoop PMC)

[jira] [Resolved] (HDFS-17204) EC: Reduce unnecessary log when processing excess redundancy.

2023-09-26 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17204. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > EC: Reduce unnecess

[jira] [Resolved] (HDFS-17197) Show file replication when listing corrupt files.

2023-09-24 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17197. Fix Version/s: 3.4.0 Hadoop Flags: Incompatible change,Reviewed Resolution: Fixed

[jira] [Resolved] (HDFS-17184) Improve BlockReceiver to throws DiskOutOfSpaceException when initialize

2023-09-21 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17184. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Improve BlockRecei

[jira] [Resolved] (HDFS-17105) mistakenly purge editLogs even after it is empty in NNStorageRetentionManager

2023-09-19 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17105. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Assignee: ConfX Resolution: Fixed

[jira] [Resolved] (HDFS-17192) Add bock info when constructing remote block reader meets IOException

2023-09-18 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17192. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add bock info w

[jira] [Resolved] (HDFS-17190) EC: Fix bug of OIV processing XAttr.

2023-09-17 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17190. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Assignee: Shuyan Zhang Resolution

[jira] [Resolved] (HDFS-17140) Revisit the BPOfferService.reportBadBlocks() method.

2023-09-06 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17140. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Revisit

[jira] [Resolved] (HDFS-16933) A race in SerialNumberMap will cause wrong owner, group and XATTR

2023-09-05 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-16933. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > A race

[jira] [Resolved] (HDFS-17093) Fix block report lease issue to avoid missing some storages report.

2023-08-27 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17093. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix block report le

[jira] [Resolved] (HDFS-17151) EC: Fix wrong metadata in BlockInfoStriped after recovery

2023-08-20 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17151. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > EC: Fix wrong metad

[jira] [Resolved] (HDFS-17154) EC: Fix bug in updateBlockForPipeline after failover

2023-08-16 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17154. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > EC: Fix bug

[jira] [Resolved] (HDFS-17150) EC: Fix the bug of failed lease recovery.

2023-08-15 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17150. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > EC: Fix the bug

[jira] [Resolved] (HDFS-17137) Standby/Observer NameNode skip to handle redundant replica block logic when set decrease replication.

2023-08-08 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17137. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Standby/Obser

[jira] [Resolved] (HDFS-17134) RBF: Fix duplicate results of getListing through Router.

2023-08-01 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17134. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Fix duplic

[jira] [Resolved] (HDFS-17116) RBF: Update invoke millisecond time as monotonicNow() in RouterSafemodeService.

2023-07-28 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17116. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Update inv

[jira] [Resolved] (HDFS-17117) Print reconstructionQueuesInitProgress periodically when BlockManager processMisReplicatesAsync.

2023-07-26 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17117. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Pr

[jira] [Resolved] (HDFS-17112) Show decommission duration in JMX and HTML

2023-07-23 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17112. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Show decommiss

[jira] [Created] (HDFS-17090) Decommission will be stuck for long time when restart because overlapped process Register and BlockReport.

2023-07-16 Thread Xiaoqiao He (Jira)
Xiaoqiao He created HDFS-17090: -- Summary: Decommission will be stuck for long time when restart because overlapped process Register and BlockReport. Key: HDFS-17090 URL: https://issues.apache.org/jira/browse/HDFS

[jira] [Resolved] (HDFS-17033) Update fsck to display stale state info of blocks accurately

2023-07-10 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17033. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Update fsck to disp

[jira] [Resolved] (HDFS-17073) Enhance the warning message output for BlockGroupNonStripedChecksumComputer#compute

2023-07-09 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17073. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Enhance the warn

Re: Fix for SEGV during java app shutfdown (HDFS-13585)

2023-07-02 Thread Xiaoqiao He
Thanks Ed for your feedback. I think it is better to comment on JIRA HDFS-13585 directly. And feel free to submit PR there if interested and there will be some reviewers coming soon. Please let me know if you will have any questions. Thanks again. Best Regards, - He Xiaoqiao On Sun, Jul 2, 2023 a

[jira] [Resolved] (HDFS-17052) Improve BlockPlacementPolicyRackFaultTolerant to avoid choose nodes failed when no enough Rack.

2023-07-02 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17052. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Committed to trunk. And

[jira] [Resolved] (HDFS-17044) Set size of non-exist block to NO_ACK when process FBR or IBR to avoid useless report from DataNode

2023-06-27 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17044. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Set size of non-ex

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

2023-06-23 Thread Xiaoqiao He
+1(binding) * Verified signature and checksum of all source tarballs. * Built source code on Ubuntu and OpenJDK 11 by `mvn clean package -DskipTests -Pnative -Pdist -Dtar`. * Setup pseudo cluster with HDFS and YARN. * Run simple FsShell - mkdir/put/get/mv/rm and check the result. * Run example mr

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

2023-06-21 Thread Xiaoqiao He
Addendum: A. Build the release from sources using: `mvn clean install package -DskipTests=true -Dmaven.javadoc.skip=true` B. It works well when using the same command to build from the source branch trunk. On Wed, Jun 21, 2023 at 8:44 PM Xiaoqiao He wrote: > Hi, > > I met a fatal e

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

2023-06-21 Thread Xiaoqiao He
Hi, I met a fatal error when building from source on local Mac OSX. It could reproduce stably. Not sure if it is related to my local environment. Try to dig it, but not any conclusion right now. Will feedback once find reasons. Appendix system environment, some more stack information refer to the

[jira] [Resolved] (HDFS-17049) EC: Fix duplicate block group IDs generated by SequentialBlockGroupIdGenerator

2023-06-20 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17049. Resolution: Not A Problem > EC: Fix duplicate block group IDs generated

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

2023-06-17 Thread Xiaoqiao He
Thanks Wei-Chiu for driving this release. The next RC will be prepared, right? If true, I would like to try and vote on the next RC. Just notice that some JIRAs are not included and need to revert some PRs to pass HBase verification which are mentioned above. Best Regards, - He Xiaoqiao On Fri,

[jira] [Resolved] (HDFS-17037) Consider nonDfsUsed when running balancer

2023-06-09 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17037. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Consider nonDfsU

[jira] [Resolved] (HDFS-17003) Erasure Coding: invalidate wrong block after reporting bad blocks from datanode

2023-06-08 Thread Xiaoqiao He (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17003. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Erasure Cod

  1   2   3   >