Re: 关于hadoop-3.3.1版本使用libhdfs3.so 访问hdfs联邦模式router节点rpc端口有20分钟延时问题

2023-10-30 Thread Xiaoqiao He
Add hdfs-dev@h.a.o and user@h.a.o

On Thu, Oct 26, 2023 at 7:07 PM 王继泽  wrote:

> 最近在使用hadoop的过程中,发现了一个情况。
> 当我使用c
> api向hdfs联邦模式router节点rpc端口发送请求时,比如说写文件,客户端发送完成请求后,hadoop端需要20分钟延时文件才有字节大小,延时期间不能对文件进行操作。
>
> 客户端这边运行结束之后,hadoop端日志大致过程:
> 1.namenode接收到客户端的请求,FSEditLog打印日志。
> 2.blockmanager.BlockPlacementPolicy: 提示没有足够的副本可供选择.
> Reason:{NO_REQUIRED_STORAGE_TYPE=1}
> 3.StateChange: 分配block
> 4.StateChange: 为hadoop目录文件获取租约
> 5.ipc.Server: 检查租约的方法抛了个异常 LeaseExpiredExcepion: INode is not a regular
> file: /
> 6.(开始等待)
> 7.20分钟后,达到hard limit最大值限制。强制关闭租约。
> 8.触发 Lease recovery
> 9.然后才可执行成功。
>
> 我也怀疑过是客户端的问题。但是我做了几组测试,(都是用c api向hadoop发送写请求,以下简写)
> 3.3.1版本、router、rpc端口。  --> 有20分钟延时
> 3.3.1版本、namenode、rpc端口。 --> 无问题
> 3.3.1版本、router、http端口。 --> 无问题
> 3.3.1版本、namenode、http端口。--> 无问题
>
> 3.1.1版本、router、rpc端口。 --> 无问题
> 3.1.1版本、namenode、rpc端口。 --> 无问题
> 3.1.1版本、router、rpc端口。 --> 无问题
> 3.1.1版本、namenode、rpc端口。 --> 无问题
>
> 以下是我的猜测:
> 从hadoop日志中看,猜测是3.3.1版本、router、rpc端口一开始未获取到租约,所以导致无法正常关闭租约,直到hard
> limit触发,才能退出。但是我无法解释为什么相同的客户端,3.1.1版本就无该现象。我怀疑是版本的变化改动与libhdfs3.so的某个部分不适配导致这一现象。
>
>
> 如果有人发现类似的情况,我希望得到回复,为我指明这个问题的方向。,
>
>
>
> | |
> 王继泽
> |
> |
> y98d...@163.com
> |
>
>


Re: request open hadoop issues to create Jira tickets

2023-02-15 Thread Xiaoqiao He
Hi Liangrui,

Please offer information as mentioned at link[1]. Thanks.

[1]
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute#HowToContribute-RequestingforaJiraaccount

Best Regards,
- He Xiaoqiao

On Wed, Feb 15, 2023 at 4:41 PM liang...@yy.com  wrote:

>
> hello
>   Can you help create a Jira ticket?I would like to open hadoop issues and
> create related issues, thank you
>
>
> liang...@yy.com
>


Re: [DISCUSS] fate of branch-2.9

2020-08-27 Thread Xiaoqiao He
+1 for putting 2.9 release lines to EOL.

Thanks,
Hexiaoqiao

On Thu, Aug 27, 2020 at 3:14 PM Mingliang Liu  wrote:

> +1 for putting 2.9 lines to EOL.
>
> Let's focus on 2.10 releases for Hadoop 2. Also is there any plan for
> 2.10.1? It has been 11 months since 2.10 first release.
>
> Thanks,
>
> On Wed, Aug 26, 2020 at 10:57 PM Wei-Chiu Chuang 
> wrote:
>
> > Bump up this thread after 6 months.
> >
> > Is anyone still interested in the 2.9 release line? Or are we good to
> start
> > the EOL process? The 2.9.2 was released in Nov 2018.
> >
> > I'd really like to see the community to converge to fewer release lines
> and
> > make more frequent releases in each line.
> >
> > Thanks,
> > Weichiu
> >
> >
> > On Fri, Mar 6, 2020 at 5:47 PM Wei-Chiu Chuang 
> wrote:
> >
> > > I think that's a great suggestion.
> > > Currently, we make 1 minor release per year, and within each minor
> > release
> > > we bring up 1 thousand to 2 thousand commits in it compared with the
> > > previous one.
> > > I can totally understand it is a big bite for users to swallow. Having
> a
> > > more frequent release cycle, plus LTS and non-LTS releases should help
> > with
> > > this. (Of course we will need to make the release preparation much
> > easier,
> > > which is currently a pain)
> > >
> > > I am happy to discuss the release model further in the dev ML. LTS v.s.
> > > non-LTS is one suggestion.
> > >
> > > Another similar issue: In the past Hadoop strived to
> > > maintain compatibility. However, this is no longer sustainable as more
> > CVEs
> > > coming from our dependencies: netty, jetty, jackson ... etc.
> > > In many cases, updating the dependencies brings breaking changes. More
> > > recently, especially in Hadoop 3.x, I started to make the effort to
> > update
> > > dependencies much more frequently. How do users feel about this change?
> > >
> > > On Thu, Mar 5, 2020 at 7:58 AM Igor Dvorzhak 
> > > wrote:
> > >
> > >> Maybe Hadoop will benefit from adopting a similar release and support
> > >> strategy as Java? I.e. designate some releases as LTS and support them
> > for
> > >> 2 (?) years (it seems that 2.7.x branch was de-facto LTS), other
> non-LTS
> > >> releases will be supported for 6 months (or until next release). This
> > >> should allow to reduce maintenance cost of non-LTS release and provide
> > >> conservative users desired stability by allowing them to wait for new
> > LTS
> > >> release and upgrading to it.
> > >>
> > >> On Thu, Mar 5, 2020 at 1:26 AM Rupert Mazzucco <
> > rupert.mazzu...@gmail.com>
> > >> wrote:
> > >>
> > >>> After recently jumping from 2.7.7 to 2.10 without issue myself, I
> vote
> > >>> for keeping only the 2.10 line.
> > >>> It would seem all other 2.x branches can upgrade to a 2.10.x easily
> if
> > >>> they feel like upgrading at all,
> > >>> unlike a jump to 3.x, which may require more planning.
> > >>>
> > >>> I also vote for having only one main 3.x branch. Why are there 3.1.x
> > and
> > >>> 3.2.x seemingly competing,
> > >>> and now 3.3.x? For a community that does not have the resources to
> > >>> manage multiple release lines,
> > >>> you guys sure like to multiply release lines a lot.
> > >>>
> > >>> Cheers
> > >>> Rupert
> > >>>
> > >>> Am Mi., 4. März 2020 um 19:40 Uhr schrieb Wei-Chiu Chuang
> > >>> :
> > >>>
> >  Forwarding the discussion thread from the dev mailing lists to the
> > user
> >  mailing lists.
> > 
> >  I'd like to get an idea of how many users are still on Hadoop 2.9.
> >  Please share your thoughts.
> > 
> >  On Mon, Mar 2, 2020 at 6:30 PM Sree Vaddi
> >   wrote:
> > 
> > > +1
> > >
> > > Sent from Yahoo Mail on Android
> > >
> > >   On Mon, Mar 2, 2020 at 5:12 PM, Wei-Chiu Chuang<
> weic...@apache.org
> > >
> > > wrote:   Hi,
> > >
> > > Following the discussion to end branch-2.8, I want to start a
> > > discussion
> > > around what's next with branch-2.9. I am hesitant to use the word
> > "end
> > > of
> > > life" but consider these facts:
> > >
> > > * 2.9.0 was released Dec 17, 2017.
> > > * 2.9.2, the last 2.9.x release, went out Nov 19 2018, which is
> more
> > > than
> > > 15 months ago.
> > > * no one seems to be interested in being the release manager for
> > 2.9.3.
> > > * Most if not all of the active Hadoop contributors are using
> Hadoop
> > > 2.10
> > > or Hadoop 3.x.
> > > * We as a community do not have the cycle to manage multiple
> release
> > > line,
> > > especially since Hadoop 3.3.0 is coming out soon.
> > >
> > > It is perhaps the time to gradually reduce our footprint in Hadoop
> > > 2.x, and
> > > encourage people to upgrade to Hadoop 3.x
> > >
> > > Thoughts?
> > >
> > >
> >
>
>
> --
> L
>