Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
For maintenance release line I also support we switch to reload4j to address the security issues first. We could file an issue for it. Andrew Purtell 于2022年1月21日 周五01:15写道： > Just to clarify: I think you want to upgrade to Log4J2 (or switch to > LogBack) as a strategy for new releases, but you have the option in > maintenance releases to use Reload4J to maintain Appender API and > operational compatibility, and users who want to minimize risks in > production while mitigating the security issues will prefer that. > > > On Jan 20, 2022, at 8:59 AM, Andrew Purtell > wrote: > > > > Reload4J has fixed all of those CVEs without requiring an upgrade. > > > >> On Jan 20, 2022, at 5:56 AM, Duo Zhang wrote: > >> > >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think > it > >> is time to speed up the migration to log4j2 work[4] now. > >> > >> You can see the discussion on the jira issue[4], our goal is to fully > >> migrate to log4j2 and the current most blocking issue is lack of the > >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already > >> started a discussion thread on the log4j dev mailing list[5] and the > result > >> is optimistic and I've filed an issue for log4j2[6], but I do not think > it > >> could be addressed and released soon. If we want to fully migrate to > >> log4j2, then either we introduce new environment variables or split the > old > >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the > >> complexity of our current startup scripts, the work is not easy and it > will > >> also break lots of other hadoop deployment systems if they do not use > our > >> startup scripts... > >> > >> So after reconsidering the current situation, I prefer we use the > log4j1.2 > >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is > >> addressed and released, we start to fully migrate to log4j2. Of course > we > >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender, > >> ContainerLogAppender and ContainerRollingLogAppender which inherit > >> FileAppender and RollingFileAppender in log4j1, which are not part of > the > >> log4j1.2 bridge. But anyway, at least we could just copy the source > code to > >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two > classes > >> do not have related CVEs. > >> > >> Thoughts? For me I would like us to make a new 3.4.x release line to > remove > >> the log4j1 dependencies ASAP. > >> > >> Thanks. > >> > >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302 > >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305 > >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307 > >> 4. https://issues.apache.org/jira/browse/HADOOP-16206 > >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3 > >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341 >
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
The EventCounter class has already been removed in HADOOP-17524. And on the filters, by default log4j1.2 bridge has log4j1 filter support, but as said above, maybe it is not fully functional if you have some advanced usage. So mind providing more information about how we use filters in Hadoop? Thanks. Arpit Agarwal 于2022年1月21日 周五00:44写道： > Hi Duo, > > Thank you for starting this discussion. Log4j1.2 bridge seems like a > practical short-term solution. However the bridge will silently affect > applications that add appenders or filters. NameNode audit logger and > metrics come to mind. There may be others. > > Thanks, > Arpit > > > > On Jan 20, 2022, at 5:55 AM, Duo Zhang wrote: > > > > There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it > > is time to speed up the migration to log4j2 work[4] now. > > > > You can see the discussion on the jira issue[4], our goal is to fully > > migrate to log4j2 and the current most blocking issue is lack of the > > "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already > > started a discussion thread on the log4j dev mailing list[5] and the > result > > is optimistic and I've filed an issue for log4j2[6], but I do not think > it > > could be addressed and released soon. If we want to fully migrate to > > log4j2, then either we introduce new environment variables or split the > old > > HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the > > complexity of our current startup scripts, the work is not easy and it > will > > also break lots of other hadoop deployment systems if they do not use our > > startup scripts... > > > > So after reconsidering the current situation, I prefer we use the > log4j1.2 > > bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is > > addressed and released, we start to fully migrate to log4j2. Of course we > > have other problems for log4j1.2 bridge too, as we have TaskLogAppender, > > ContainerLogAppender and ContainerRollingLogAppender which inherit > > FileAppender and RollingFileAppender in log4j1, which are not part of the > > log4j1.2 bridge. But anyway, at least we could just copy the source code > to > > hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes > > do not have related CVEs. > > > > Thoughts? For me I would like us to make a new 3.4.x release line to > remove > > the log4j1 dependencies ASAP. > > > > Thanks. > > > > 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302 > > 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305 > > 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307 > > 4. https://issues.apache.org/jira/browse/HADOOP-16206 > > 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3 > > 6. https://issues.apache.org/jira/browse/LOG4J2-3341 > > > - > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
[DISCUSS] Migrate hadoop from log4j1 to log4j2
There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it is time to speed up the migration to log4j2 work[4] now. You can see the discussion on the jira issue[4], our goal is to fully migrate to log4j2 and the current most blocking issue is lack of the "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already started a discussion thread on the log4j dev mailing list[5] and the result is optimistic and I've filed an issue for log4j2[6], but I do not think it could be addressed and released soon. If we want to fully migrate to log4j2, then either we introduce new environment variables or split the old HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the complexity of our current startup scripts, the work is not easy and it will also break lots of other hadoop deployment systems if they do not use our startup scripts... So after reconsidering the current situation, I prefer we use the log4j1.2 bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is addressed and released, we start to fully migrate to log4j2. Of course we have other problems for log4j1.2 bridge too, as we have TaskLogAppender, ContainerLogAppender and ContainerRollingLogAppender which inherit FileAppender and RollingFileAppender in log4j1, which are not part of the log4j1.2 bridge. But anyway, at least we could just copy the source code to hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes do not have related CVEs. Thoughts? For me I would like us to make a new 3.4.x release line to remove the log4j1 dependencies ASAP. Thanks. 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307 4. https://issues.apache.org/jira/browse/HADOOP-16206 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3 6. https://issues.apache.org/jira/browse/LOG4J2-3341
Re: [jira] [Created] (HADOOP-18069) CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client
https://square.github.io/okhttp/changelog/ The latest stable release is 4.9.1 which was published at 2021.1.30 https://github.com/square/okhttp/commits/master And there are still lots of commits recently. I'm not saying we should not remove it in hadoop, just want to point out that it is still under development and maintenance... Thanks. Steve Loughran 于2022年1月7日周五 22:40写道： > > okhttp was last updated in 2017 > > why use this over httpclient? its only used in a couple of places and > removing it entirely would make this problem go away forever > > -- Forwarded message - > From: Eugene Shinn (Truveta) (Jira) > Date: Wed, 5 Jan 2022 at 19:48 > Subject: [jira] [Created] (HADOOP-18069) CVE-2021-0341 in okhttp@2.7.5 > detected in hdfs-client > To: > > > Eugene Shinn (Truveta) created HADOOP-18069: > --- > > Summary: CVE-2021-0341 in okhttp@2.7.5 detected in > hdfs-client > Key: HADOOP-18069 > URL: https://issues.apache.org/jira/browse/HADOOP-18069 > Project: Hadoop Common > Issue Type: Bug > Components: hdfs-client > Affects Versions: 3.3.1 > Reporter: Eugene Shinn (Truveta) > > > Our static vulnerability scanner (Fortify On Demand) detected [NVD - > CVE-2021-0341 (nist.gov)| > https://nvd.nist.gov/vuln/detail/CVE-2021-0341#VulnChangeHistorySection] in > our application. We traced the vulnerability to a transitive dependency > coming from hadoop-hdfs-client, which depends on okhttp@2.7.5 > ([hadoop/pom.xml at trunk · apache/hadoop (github.com)| > https://github.com/apache/hadoop/blob/trunk/hadoop-project/pom.xml#L137]). > To resolve this issue, okhttp should be upgraded to 4.9.2+ (ref: > [CVE-2021-0341 · Issue #6724 · square/okhttp (github.com)| > https://github.com/square/okhttp/issues/6724]). > > > > -- > This message was sent by Atlassian Jira > (v8.20.1#820001) > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Change project style guidelines to allow line length 100
+1 for 100 instead of 80. I can see that some projects have already changed to 120. Akira Ajisaka 于2021年5月20日周四 上午10:01写道： > I'm +1 to allow <= 100 chars. > > FYI: There were some discussions long before: > - > https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E > - > https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E > > Thanks, > Akira > > On Thu, May 20, 2021 at 6:36 AM Sean Busbey > wrote: > > > > Hello! > > > > What do folks think about changing our line length guidelines to allow > for 100 character width? > > > > Currently, we tell folks to follow the sun style guide with some > exception unrelated to line length. That guide says width of 80 is the > standard and our current check style rules act as enforcement. > > > > Looking at the current trunk codebase our nightly build shows a total of > ~15k line length violations; it’s about 18% of identified checkstyle issues. > > > > The vast majority of those line length violations are <= 100 characters > long. 100 characters happens to be the length for the Google Java Style > Guide, another commonly adopted style guide for java projects, so I suspect > these longer lines leaking past the checkstyle precommit warning might be a > reflection of committers working across multiple java codebases. > > > > I don’t feel strongly about lines being longer, but I would like to move > towards more consistent style enforcement as a project. Updating our > project guidance to allow for 100 character lines would reduce the > likelihood that folks bringing in new contributions need a precommit test > cycle to get the formatting correct. > > > > Does anyone feel strongly about keeping the line length limit at 80 > characters? > > > > Does anyone feel strongly about contributions coming in that clear up > line length violations? > > > > > > - > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
Re: Status of Hadoop 2.9.x
Yes, HBase is proposing to bump hadoop versions for new release lines, so we want to know which release lines are still alive for hadoop. Thanks. Wei-Chiu Chuang 于2019年12月26日周四 下午6:59写道： > Hi! > > Duo found out that the last 2.9.x release, 2.9.2 was GA in Nov 2018, which > is a little more than a year ago. Are we still interested in maintaining > that branch? Any volunteer likes to continue it? As far as I know, Twitter > just recently completed upgrade to 2.9 so I imagine there are still people > who want to maintain it. > > https://hadoop.apache.org/releases.html >
Re: [DISCUSS] Remove Ozone and Submarine from Hadoop repo
+1 Akira Ajisaka 于2019年10月24日周四 下午3:21写道： > Hi folks, > > Both Ozone and Apache Submarine have separate repositories. > Can we remove these modules from hadoop-trunk? > > Regards, > Akira >
Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts
For HBase we have a separated repo for hbase-thirdparty https://github.com/apache/hbase-thirdparty We will publish the artifacts to nexus so we do not need to include binaries in our git repo, just add a dependency in the pom. https://mvnrepository.com/artifact/org.apache.hbase.thirdparty/hbase-shaded-protobuf And it has its own release cycles, only when there are special requirements or we want to upgrade some of the dependencies. This is the vote thread for the newest release, where we want to provide a shaded gson for jdk7. https://lists.apache.org/thread.html/f12c589baabbc79c7fb2843422d4590bea982cd102e2bd9d21e9884b@%3Cdev.hbase.apache.org%3E Thanks. Vinayakumar B 于2019年9月28日周六 上午1:28写道： > Please find replies inline. > > -Vinay > > On Fri, Sep 27, 2019 at 10:21 PM Owen O'Malley > wrote: > > > I'm very unhappy with this direction. In particular, I don't think git is > > a good place for distribution of binary artifacts. Furthermore, the PMC > > shouldn't be releasing anything without a release vote. > > > > > Proposed solution doesnt release any binaries in git. Its actually a > complete sub-project which follows entire release process, including VOTE > in public. I have mentioned already that release process is similar to > hadoop. > To be specific, using the (almost) same script used in hadoop to generate > artifacts, sign and deploy to staging repository. Please let me know If I > am conveying anything wrong. > > > > I'd propose that we make a third party module that contains the *source* > > of the pom files to build the relocated jars. This should absolutely be > > treated as a last resort for the mostly Google projects that regularly > > break binary compatibility (eg. Protobuf & Guava). > > > > > Same has been implemented in the PR > https://github.com/apache/hadoop-thirdparty/pull/1. Please check and let > me > know If I misunderstood. Yes, this is the last option we have AFAIK. > > > > In terms of naming, I'd propose something like: > > > > org.apache.hadoop.thirdparty.protobuf2_5 > > org.apache.hadoop.thirdparty.guava28 > > > > In particular, I think we absolutely need to include the version of the > > underlying project. On the other hand, since we should not be shading > > *everything* we can drop the leading com.google. > > > > > IMO, This naming convention is easy for identifying the underlying project, > but it will be difficult to maintain going forward if underlying project > versions changes. Since thirdparty module have its own releases, each of > those release can be mapped to specific version of underlying project. Even > the binary artifact can include a MANIFEST with underlying project details > as per Steve's suggestion on HADOOP-13363. > That said, if you still prefer to have project number in artifact id, it > can be done. > > The Hadoop project can make releases of the thirdparty module: > > > > > > org.apache.hadoop > > hadoop-thirdparty-protobuf25 > > 1.0 > > > > > > > Note that the version has to be the hadoop thirdparty release number, which > > is part of why you need to have the underlying version in the artifact > > name. These we can push to maven central as new releases from Hadoop. > > > > > Exactly, same has been implemented in the PR. hadoop-thirdparty module have > its own releases. But in HADOOP Jira, thirdparty versions can be > differentiated using prefix "thirdparty-". > > Same solution is being followed in HBase. May be people involved in HBase > can add some points here. > > Thoughts? > > > > .. Owen > > > > On Fri, Sep 27, 2019 at 8:38 AM Vinayakumar B > > wrote: > > > >> Hi All, > >> > >>I wanted to discuss about the separate repo for thirdparty > dependencies > >> which we need to shaded and include in Hadoop component's jars. > >> > >>Apologies for the big text ahead, but this needs clear explanation!! > >> > >>Right now most needed such dependency is protobuf. Protobuf > dependency > >> was not upgraded from 2.5.0 onwards with the fear that downstream > builds, > >> which depends on transitive dependency protobuf coming from hadoop's > jars, > >> may fail with the upgrade. Apparently protobuf does not guarantee source > >> compatibility, though it guarantees wire compatibility between versions. > >> Because of this behavior, version upgrade may cause breakage in known > and > >> unknown (private?) downstreams. > >> > >>So to tackle this, we came up the following proposal in HADOOP-13363. > >> > >>Luckily, As far as I know, no APIs, either public to user or between > >> Hadoop processes, is not directly using protobuf classes in signatures. > >> (If > >> any exist, please let us know). > >> > >>Proposal: > >> > >> > >>1. Create a artifact(s) which contains shaded dependencies. All such > >> shading/relocation will be with known prefix > >> **org.apache.hadoop.thirdparty.**. > >>2. Right now protobuf jar (ex: > o.a.h.thirdparty:hadoop-shaded-protobuf) > >> to start with, all **com.googl
Re: [NOTICE] Building trunk needs protoc 3.7.1
The new protobuf plugin related issues have all been pushed to trunk(though I think we'd better port them to all active branches). So what's the next step? Shade and relocated protobuf? HBase has already done this before so I do not think it will take too much time. If we all agree on the solution, I think we can finish this in one week. But maybe a problem is that, is it OK to upgrade protobuf in a minor release? Of course if we shade and relocate protobuf it will less hurt to users as they can depend on protobuf 2.5 explicitly if they want, but still a bit uncomfortable. Thanks. Wangda Tan 于2019年9月24日周二 上午2:29写道： > Hi Vinay, > > Thanks for the clarification. > > Do you have a timeline about all you described works w.r.t. the > compatibility will be completed? I'm asking this is because we need to > release 3.3.0 earlier if possible since there're 1k+ of patches in 3.3.0 > already, we should get it out earlier. > > If the PB work will take more time, do you think if we should create a > branch for 3.3, revert PB changes from branch-3.3, and keep on working on > PB for the next minor release? (or major release if we do see some > compatibility issue in the future). > > Just my $0.02 > > Thanks, > Wangda > > On Mon, Sep 23, 2019 at 5:43 AM Steve Loughran > > wrote: > > > aah, that makes sense > > > > On Sun, Sep 22, 2019 at 6:11 PM Vinayakumar B > > wrote: > > > > > Thanks Steve. > > > > > > Idea is not to shade all artifacts. > > > Instead maintain one artifact ( hadoop-thirdparty) which have all such > > > dependencies ( com.google.* may be), add this artifact as dependency > in > > > hadoop modules. Use shaded classes directly in the code of hadoop > modules > > > instead of shading at package phase. > > > > > > Hbase, ozone and ratis already following this way. The artifact ( > > > hadoop-thirdparty) with shaded dependencies can be maintained in a > > separate > > > repo as suggested by stack on HADOOP-13363 or could be maintained as a > > > separate module in Hadoop repo. If maintained in separate repo, need to > > > build this only when there are changes related to shaded dependencies. > > > > > > > > > -Vinay > > > > > > On Sun, 22 Sep 2019, 10:11 pm Steve Loughran, > > wrote: > > > > > > > > > > > > > > > On Sun, Sep 22, 2019 at 3:22 PM Vinayakumar B < > vinayakum...@apache.org > > > > > > > wrote: > > > > > > > >>Protobuf provides Wire compatibility between releases.. but not > > > >> guarantees the source compatibility in generated sources. There will > > be > > > a > > > >> problem in compatibility if anyone uses generated protobuf message > > > outside > > > >> of Hadoop modules. Which ideally shouldn't be as generated sources > are > > > not > > > >> public APIs. > > > >> > > > >>There should not be any compatibility problems between releases > in > > > >> terms > > > >> of communication provided both uses same syntax (proto2) of proto > > > message. > > > >> This I have verified by communication between protobuf 2.5.0 client > > with > > > >> protobuf 3.7.1 server. > > > >> > > > >>To avoid the downstream transitive dependency classpath problem, > > who > > > >> might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 > > > classes > > > >> and its usages in all hadoop modules.. and keep 2.5.0 jar back in > > hadoop > > > >> classpath. > > > >> > > > >> Hope I have answered your question. > > > >> > > > >> -Vinay > > > >> > > > >> > > > > While I support the move and CP isolation, this is going to (finally) > > > > force us to make shaded versions of all artifacts which we publish > with > > > the > > > > intent of them being loaded on the classpath of other applications > > > > > > > > > >
Re: [NOTICE] Building trunk needs protoc 3.7.1
I think this one is alread in place so we have to upgrade... https://issues.apache.org/jira/browse/HADOOP-16557 Wangda Tan 于2019年9月21日周六 上午7:19写道： > Hi Vinay, > > A bit confused, I saw the HADOOP-13363 is still pending. Do we need to > upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is completed? > > Thanks, > Wangda > > On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B > wrote: > > > Hi All, > > > > A very long pending task, protobuf upgrade is happening in HADOOP-13363. > As > > part of that protobuf version is upgraded to 3.7.1. > > > > Please update your build environments to have 3.7.1 protobuf version. > > > > BUILIDING.txt has been updated with latest instructions. > > > > This pre-requisite to update protoc dependecy manually is required until > > 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to > > dynamically resolve required protoc exe. > > > > Dockerfile is being updated to have latest 3.7.1 as default protoc for > test > > environments. > > > > Thanks, > > -Vinay > > >
Re: [DISCUSS] GitHub PRs without JIRA number
OK, the indexing is finished [Tue Sep 10 14:25:07 UTC 2019] Finished branch indexing. Indexing took 15 min So maybe 30 minutes is better... And we really should cleanup the stale PRs and branches... 张铎(Duo Zhang) 于2019年9月10日周二 下午10:20写道： > Actually the job for testing PR is here... > > https://builds.apache.org/job/hadoop-multibranch/ > > I've added the 'Change requests' option(seems I have the permission...), > and then clicked the 'Scan Now', the job for PR-1404 has been scheduled > > > https://builds.apache.org/job/hadoop-multibranch/view/change-requests/job/PR-1404/ > > > The infra team has shutdown the github webhook on scheduling builds > automatically when there are new PRs or new updates to existing PRs, > because this violate the rule that only committers can trigger jenkins > jobs, so we need to click the 'Scan Now' button to trigger builds. And it > is possible to schedule a periodical job to do the scan, for HBase the > interval is 10 minutes, and for hadoop, there are too many branches and > PRs, the job is still running while I'm writing this email, so I think > maybe we should use a greater interval, maybe 20 or 30 minutes? > > Thanks. > > Steve Loughran 于2019年9月10日周二 下午7:36写道： > >> >> >> On Tue, Sep 10, 2019 at 9:07 AM 张铎(Duo Zhang) >> wrote: >> >>> Nits: You can change the jenkins job config to not trigger pre commit >>> build >>> for stale PRs if only the base branch is changed. By default, either the >>> PR >>> itself changed, or the base branch is changed, the branch sources plugin >>> will both trigger a build. You can add a Change requests option, and >>> select >>> 'Ignore rebuilding merge branches when only the target branch changed'. >>> So >>> the stale PRs will not waste jenkins build resources any more. >>> >>> +1 for this. If old PRs want to be tested by their creator, they can >> rebase. Having a way to ask jenkins to do it on the PR lets other. >> >> >>> And on retesting a PR, just go to the pipeline jenkins job, find the >>> related job for the PR, and click build manually. >>> >>> >>> >> >>1. I like the command approach. Spark has this, and some gerrit >>pipelines do. >>2. I've just tried to do this on a PR ( >>https://github.com/apache/hadoop/pull/1404) but can't see how to. The >>precommit hdfs job only takes the JIRA number and looks for an attachment, >>judging by the way it is failing to merge things in >> >> https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-HDFS-Build/27830/ >> >> Is there another build or some trick I should use? >> Ni >> >
Re: [DISCUSS] GitHub PRs without JIRA number
Actually the job for testing PR is here... https://builds.apache.org/job/hadoop-multibranch/ I've added the 'Change requests' option(seems I have the permission...), and then clicked the 'Scan Now', the job for PR-1404 has been scheduled https://builds.apache.org/job/hadoop-multibranch/view/change-requests/job/PR-1404/ The infra team has shutdown the github webhook on scheduling builds automatically when there are new PRs or new updates to existing PRs, because this violate the rule that only committers can trigger jenkins jobs, so we need to click the 'Scan Now' button to trigger builds. And it is possible to schedule a periodical job to do the scan, for HBase the interval is 10 minutes, and for hadoop, there are too many branches and PRs, the job is still running while I'm writing this email, so I think maybe we should use a greater interval, maybe 20 or 30 minutes? Thanks. Steve Loughran 于2019年9月10日周二 下午7:36写道： > > > On Tue, Sep 10, 2019 at 9:07 AM 张铎(Duo Zhang) > wrote: > >> Nits: You can change the jenkins job config to not trigger pre commit >> build >> for stale PRs if only the base branch is changed. By default, either the >> PR >> itself changed, or the base branch is changed, the branch sources plugin >> will both trigger a build. You can add a Change requests option, and >> select >> 'Ignore rebuilding merge branches when only the target branch changed'. So >> the stale PRs will not waste jenkins build resources any more. >> >> +1 for this. If old PRs want to be tested by their creator, they can > rebase. Having a way to ask jenkins to do it on the PR lets other. > > >> And on retesting a PR, just go to the pipeline jenkins job, find the >> related job for the PR, and click build manually. >> >> >> > >1. I like the command approach. Spark has this, and some gerrit >pipelines do. >2. I've just tried to do this on a PR ( >https://github.com/apache/hadoop/pull/1404) but can't see how to. The >precommit hdfs job only takes the JIRA number and looks for an attachment, >judging by the way it is failing to merge things in > > https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-HDFS-Build/27830/ > > Is there another build or some trick I should use? > Ni >
Re: [DISCUSS] GitHub PRs without JIRA number
Nits: You can change the jenkins job config to not trigger pre commit build for stale PRs if only the base branch is changed. By default, either the PR itself changed, or the base branch is changed, the branch sources plugin will both trigger a build. You can add a Change requests option, and select 'Ignore rebuilding merge branches when only the target branch changed'. So the stale PRs will not waste jenkins build resources any more. And on retesting a PR, just go to the pipeline jenkins job, find the related job for the PR, and click build manually. Thanks. Akira Ajisaka 于2019年9月10日周二 下午2:30写道： > Thanks all for the discussion. > > I'm +1 for adding PR template and I wrote a patch long ago: > https://issues.apache.org/jira/browse/HADOOP-15184 > > I'm interested in "test this", "retest this", etc. > I'll file a jira for this feature. > > -Akira > > On Thu, Sep 5, 2019 at 4:05 AM Sean Busbey > wrote: > > > We should add a Pull Request Template that specifically calls out the > > expectation that folks need to have a JIRA associated with their PR > > for it to get reviewed. Expectations around time to response and how > > to go about getting attention when things lag would also be good to > > include. (e.g. are folks expected to ping on the jira? are folks > > expected to email a relevant *-dev list?) > > > > If anyone is interested in doing the work to make it so "test this" / > > "retest this" / etc work, open a jira and I'll give you some pointers > > of examples to go off of. We use a plugin to do this for yetus based > > tests in some HBase repos. > > > > On Wed, Sep 4, 2019 at 1:59 PM Wei-Chiu Chuang > > wrote: > > > > > > +general@ > > > > > > > > > On Wed, Aug 28, 2019 at 6:42 AM Wei-Chiu Chuang > > wrote: > > > > > > > I don't think our GitHub integration supports those commands. Ozone > has > > > > its own github integration that can test individual PRs though. > > > > > > > > > > > > > > > > On Tue, Aug 27, 2019 at 12:40 PM Iñigo Goiri > > wrote: > > > > > > > >> I wouldn't go for #3 and always require a JIRA for a PR. > > > >> > > > >> In general, I think we should state the best practices for using > > GitHub > > > >> PRs. > > > >> There were some guidelines but they were kind of open > > > >> For example, adding always a link to the JIRA to the description. > > > >> I think PRs can have a template as a start. > > > >> > > > >> The other thing I would do is to disable the automatic Jenkins > > trigger. > > > >> I've seen the "retest this" and others: > > > >> > > > https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin > > > >> https://github.com/jenkinsci/ghprb-plugin/blob/master/README.md > > > >> > > > >> > > > >> > > > >> On Tue, Aug 27, 2019 at 10:47 AM Wei-Chiu Chuang < > weic...@apache.org> > > > >> wrote: > > > >> > > > >> > Hi, > > > >> > There are hundreds of GitHub PRs pending review. Many of them just > > sit > > > >> > there wasting Jenkins resources. > > > >> > > > > >> > I suggest: > > > >> > (1) close PRs that went stale (i.e. doesn't compile). Or even > close > > PRs > > > >> > that hasn't been reviewed for more than a year. > > > >> > (1) close PRs that doesn't have a JIRA number. No one is going to > > > >> review a > > > >> > big PR that doesn't have a JIRA anyway. > > > >> > (2) For PRs without JIRA number, file JIRAs for the PR on behalf > of > > the > > > >> > reporter. > > > >> > (3) For typo fixes, merge the PRs directly without a JIRA. IMO, > > this is > > > >> the > > > >> > best use of GitHub PR. > > > >> > > > > >> > Thoughts? > > > >> > > > > >> > > > > > > > > > > > > -- > > busbey > > > > - > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > >
Re: [DISCUSS] ARM/aarch64 support for Hadoop
Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate the protobuf code. It will download the protoc binary from the maven central so we do not need to install protoc on the build machine any more. Zhenyu Zheng 于2019年9月4日周三 下午5:27写道： > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling for > over 2 month related to the Protobuf problem . > According to the latest successful build log: > https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull > the > os version was ubuntu 14.04 and for the jobs that are failling now such > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console, > the os version is 18.04. I'm not very familiar with the version changing > for the jobs but I did a little search, according to: > > https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names > & > > https://packages.ubuntu.com/search?suite=default§ion=all&arch=any&keywords=libprotoc-dev&searchon=names > it both said that the version of libprotc-dev and protobuf-compiler > available for ubuntu 18.04 is 3.0.0 > > > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena wrote: > >> Thanx Vinay for the initiative, Makes sense to add support for different >> architectures. >> >> +1, for the branch idea. >> Good Luck!!! >> >> -Ayush >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) >> wrote: >> > >> > For HBase, we purged all the protobuf related things from the public >> API, >> > and then upgraded to a shaded and relocated version of protobuf. We have >> > created a repo for this: >> > >> > https://github.com/apache/hbase-thirdparty >> > >> > But since the hadoop dependencies still pull in the protobuf 2.5 jars, >> our >> > coprocessors are still on protobuf 2.5. Recently we have opened a >> discuss >> > on how to deal with the upgrading of coprocessor. Glad to see that the >> > hadoop community is also willing to solve the problem. >> > >> > Anu Engineer 于2019年9月3日周二 上午1:23写道： >> > >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving that >> >> Hadoop and the downstream projects work correctly after you upgrade >> core >> >> components like Protobuf. >> >> So while branching and working on a branch is easy, merging back after >> you >> >> upgrade some of these core components is insanely hard. You might want >> to >> >> make sure that community buys into upgrading these components in the >> trunk. >> >> That way we will get testing and downstream components will notice when >> >> things break. >> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a really long >> >> time; I have argued that 2.5 is out of support and we cannot stay on >> that >> >> branch forever; or we need to take ownership of the Protobuf 2.5 code >> base. >> >> It has been rightly pointed to me that while all the arguments I make >> is >> >> correct; it is a very complicated task to upgrade Protobuf, and the >> worst >> >> part is we will not even know what breaks until downstream projects >> pick up >> >> these changes and work against us. >> >> >> >> If we work off the Hadoop version 3 — and assume that we have >> "shading" in >> >> place for all deployments; it might be possible to get there; still a >> >> daunting task. >> >> >> >> So best of luck with the branch approach — But please remember, Merging >> >> back will be hard, Just my 2 cents. >> >> >> >> — Anu >> >> >> >> >> >> >> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng > > >> >> wrote: >> >> >> >>> Hi, >> >>> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A >> separate >> >>> branch with it's own ARM CI seems a really good idea. >> >>> By doing this we won't break any of the undergoing development in >> trunk >> >> and >> >>> a CI can be a very good way to show what are the >> >>> current problems and what have been fixed, it will also provide a very >> >> good >> >>> view for contributors that are intrested to working on >> >>> this. We can finally merge back the branch to trunk until the >> community >> >>>
Re: [DISCUSS] ARM/aarch64 support for Hadoop
For HBase, we purged all the protobuf related things from the public API, and then upgraded to a shaded and relocated version of protobuf. We have created a repo for this: https://github.com/apache/hbase-thirdparty But since the hadoop dependencies still pull in the protobuf 2.5 jars, our coprocessors are still on protobuf 2.5. Recently we have opened a discuss on how to deal with the upgrading of coprocessor. Glad to see that the hadoop community is also willing to solve the problem. Anu Engineer 于2019年9月3日周二 上午1:23写道： > +1, for the branch idea. Just FYI, Your biggest problem is proving that > Hadoop and the downstream projects work correctly after you upgrade core > components like Protobuf. > So while branching and working on a branch is easy, merging back after you > upgrade some of these core components is insanely hard. You might want to > make sure that community buys into upgrading these components in the trunk. > That way we will get testing and downstream components will notice when > things break. > > That said, I have lobbied for the upgrade of Protobuf for a really long > time; I have argued that 2.5 is out of support and we cannot stay on that > branch forever; or we need to take ownership of the Protobuf 2.5 code base. > It has been rightly pointed to me that while all the arguments I make is > correct; it is a very complicated task to upgrade Protobuf, and the worst > part is we will not even know what breaks until downstream projects pick up > these changes and work against us. > > If we work off the Hadoop version 3 — and assume that we have "shading" in > place for all deployments; it might be possible to get there; still a > daunting task. > > So best of luck with the branch approach — But please remember, Merging > back will be hard, Just my 2 cents. > > — Anu > > > > > On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng > wrote: > > > Hi, > > > > Thanks Vinaya for bring this up and thanks Sheng for the idea. A separate > > branch with it's own ARM CI seems a really good idea. > > By doing this we won't break any of the undergoing development in trunk > and > > a CI can be a very good way to show what are the > > current problems and what have been fixed, it will also provide a very > good > > view for contributors that are intrested to working on > > this. We can finally merge back the branch to trunk until the community > > thinks it is good enough and stable enough. We can donate > > ARM machines to the existing CI system for the job. > > > > I wonder if this approch possible? > > > > BR, > > > > On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu > wrote: > > > > > Hi, > > > > > > Thanks Vinay for bring this up, I am a member of "Openlab" community > > > mentioned by Vinay. I am working on building and > > > testing Hadoop components on aarch64 server these days, besides the > > missing > > > dependices of ARM platform issues #1 #2 #3 > > > mentioned by Vinay, other similar issue has also be found, such as the > > > "PhantomJS" dependent package also missing for aarch64. > > > > > > To promote the ARM support for Hadoop, we have discussed and hoped to > add > > > an ARM specific CI to Hadoop repo. we are not > > > sure about if there is any potential effect or confilict on the trunk > > > branch, so maybe creating a ARM specific branch for doing these stuff > > > is a better choice, what do you think? > > > > > > Hope to hear thoughts from you :) > > > > > > BR, > > > Liu sheng > > > > > > Vinayakumar B 于2019年8月27日周二 上午5:34写道： > > > > > > > Hi Folks, > > > > > > > > ARM is becoming famous lately in its processing capability and has > got > > > the > > > > potential to run Bigdata workloads. > > > > Many users have been moving to ARM machines due to its low cost. > > > > > > > > In the past there were attempts to compile Hadoop on ARM (Rasberry > PI) > > > for > > > > experimental purposes. Today ARM architecture is taking some of the > > > > serverside processing as well. So there will be/is a real need of > > Hadoop > > > to > > > > support ARM architecture as well. > > > > > > > > There are bunch of users who are trying out building Hadoop on ARM, > > > trying > > > > to add ARM CI to hadoop and facing issues[1]. Also some > > > > > > > > As of today, Hadoop does not compile on ARM due to below issues, > found > > > from > > > > testing done in openlab in [2]. > > > > > > > > 1. Protobuf : > > > > --- > > > > Hadoop project (also some downstream projects) stuck to protobuf > > > 2.5.0 > > > > version, due to backward compatibility reasons. Protobuf-2.5.0 is not > > > being > > > > maintained in the community. While protobuf 3.x is being actively > > adopted > > > > widely, still protobuf 3.x provides wire compatibility for proto2 > > > messages. > > > > Due to some compilation issues in the generated java code, which can > > > induce > > > > problems in downstream. Due to this reason protobuf upgrade from > 2.5.0 > > > was > > > > not taken up. > > > > In 3.0.0 onward
Re: [DISCUSS] EOL 2.8 or another 2.8.x release?
IIRC we have a 2.8.5 release? On the download page: 2.8.5 2018 Sep 15 Wei-Chiu Chuang 于2019年7月25日周四 上午9:39写道： > The last 2.8 release (2.8.4) was made in the last May, more than a year > ago. https://hadoop.apache.org/old/releases.html > > How do folks feel about the fate of branch-2.8? During the last community > meetup in June, it sounds like most users are still on 2.8 or even 2.7, so > I don't think we want to abandon 2.8 just yet. > > I would personally want to urge folks to move up to 3.x, so I can stop > cherrypicking stuff all the way down into 2.8. But it's not up to me along > to decide :) > > How do people feel about having another 2.8 release or two? I am not saying > I want to drive it, but I want to raise the awareness that folks are still > on 2.8 and there's not been an update for over a year. > > Thoughts? >
Re: Please help to review patch of the HDFS-14585 (backport Byte
Thanks for your reply Sree Vaddi. As you said, in our internal hadoop branch we have already backported these changes, and also it is fine for us to upgrade to hadoop 3, in fact we have already started to test hadoop 3.1.x in our company. But here we are talking about the open source version of hbase being affected by these changes. For the open source version, I do not think we can drop all the hadoop 2.x support for hbase 2.x releases. We have to follow some rules, for example, only drop hadoop support when there are CVEs and the hadoop team will not fix it for a minor release line. And when changing the major versions, we can drop some legacy hadoop support, for example, when releasing hbase 3.x, probably we could drop all the support for hadoop 2.x. Thanks. Regards OpenInx 于2019年6月28日周五 上午10:58写道： > > What are your challenges upgrading to Hadoop 3 ? > > It's a major release upgradation from hadoop 2 to hadoop 3. we upstreams > will need some time to test the stability > and performance, so I think the HBase will not upgrade to hadoop 3 so quick > in short time. Backporting the ByteBuffer > pread to hadoop 2.9 is more suitable for now (it's a minor feature, should > be OK to port in branch-2.9). I believe in future > HBase release we will include the hadoop 3 > > > Did you verify if your problem statement valid, if you use Ozone API ? > We have some work about the object-store api [1]. but I'm not quite sure > how will it affect the GC. will take a look about that. > > Thanks for your reply. > > [1]. https://issues.apache.org/jira/browse/HBASE-22149 > On Fri, Jun 28, 2019 at 7:28 AM Sree Vaddi > wrote: > > > What are your challenges upgrading to Hadoop 3 ? > > > > Did you verify if your problem statement valid, if you use Ozone API ? > > > > Sent from Yahoo Mail on Android > > < > https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature > > > > > > On Thu, Jun 27, 2019 at 9:20 AM, OpenInx > > wrote: > > Dear HDFS Team: > > > > I'm apache HBase member and working on HBASE-21879 [1]: it's a feature > > which help to optimize the > > HBase GC issues, we redesigned the HBase read(Scan/Get) path by reading > > HDFS block into offheap > > ByteBuffers and it had great performance improvement in our benchmark > test > > [2]. While the offheap > > HDFS block reading highly depend on the HDFS ByteBuffer interfaces, such > > as: ByteBuffer stream > > read and ByteBuffer positional read. > > > > The problem is ByteBuffer pread interface: we've finished the > HDFS-3246[3] > > for Hadoop 3.3.0 version. but > > the Apache HBase is depending on Hadoop 2.8.5 version, and seems will not > > upgrade the dependency to > > Hadoop 3.3.0 soon. We have to backport HDFS-3246 to Hadoop branch-2.9 > so > > that in the next hadoop > > 2.9.3 can include it , and our Apache HBase can upgrade the dependency to > > 2.9.3. > > > > Now all the patches of HBASE-21879 have been merged into HBase master > > branch. If we cannot upgrade to > > the Hadoop which include the ByteBuffer pread implementation, then I > > believe almost all of work from HBASE-21879 > > will worth nothing because we still read heap block and still high GC > > pressure in some special case ( such as > > cacheHitRatio is not very high). That will be quite frustrating for us. > > > > Actually, we have been pushing the backport for the past few weeks. our > > XiaoMi HDFS team have prepared the > > backport patches, but seems the guys we ping didn't have the bandwidth to > > help review the patches. we > > expect the backport can be included in Hadoop 2.9.3 so that our upstream > > can benefit from this as soon as > > possible. > > > > If any HDFS committer or PMC have some bandwidth to review the backport > > patches, please give us a hand. > > The Apache HBase team really need your help. > > > > The ByteBuffer pread backport issues are: > > https://issues.apache.org/jira/browse/HDFS-14585 > > https://issues.apache.org/jira/browse/HDFS-14483 > > > > Thank you very much. > > > > [1]. https://issues.apache.org/jira/browse/HBASE-21879 > > [2]. https://www.mail-archive.com/dev@hbase.apache.org/msg55335.html > > [3]. https://issues.apache.org/jira/browse/HDFS-3246 > > > > >
Re: Jenkins precommit build for HDFS failing
Solved by https://issues.apache.org/jira/browse/INFRA-17068. 张铎(Duo Zhang) 于2018年9月28日周五 上午9:37写道： > I tried to upgrade the yetus version for hbase pre commit job and the > error message is the same with hadoop then. And the problem is that when > fetching the patch we get a 4xx, so we just apply a nonexistent file... > > jira_http_fetch: https://issues.apache.org/jira/browse/HBASE-21233 returned > 4xx status code. Maybe incorrect username/password? > > > Ted Yu 于2018年9月28日周五 上午6:20写道： > >> Over in hbase precommit, I saw this: >> https://builds.apache.org/job/PreCommit-HBASE-Build/14514/console >> >> Resolving deltas: 86% (114758/133146), completed with 87 local >> objects.*21:06:47* fatal: pack has 18388 unresolved deltas*21:06:47* >> fatal: index-pack failed*21:06:47* *21:06:47* at >> >> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)*21:06:47* >> at >> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1721)*21:06:47* >> at >> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72) >> >> >> It seems QA machine(s) may have trouble accessing git. >> >> >> I wonder if the 'index-pack failed' error would lead to patch not >> being recognized. >> >> >> FYI >> >> >> On Thu, Sep 27, 2018 at 3:02 PM Ajay Kumar >> wrote: >> >> > Hi, >> > >> > Jenkins precommit build for HDFS is failing with error that patch >> doesn’t >> > apply to trunk, even when patch applies to trunk. >> > I see other build failures with same error for other patches as well. >> > Wanted to reach out to know if it is a known issue. >> > >> > >> > Vote >> > >> > Subsystem >> > >> > Runtime >> > >> > Comment >> > >> > 0 >> > >> > reexec >> > >> > 0m 0s >> > >> > Docker mode activated. >> > >> > -1 >> > >> > patch >> > >> > 0m 5s >> > >> > HDFS-13941 does not apply to trunk. Rebase required? Wrong Branch? See >> > https://wiki.apache.org/hadoop/HowToContribute for help. >> > >> > >> > Subsystem >> > >> > Report/Notes >> > >> > JIRA Issue >> > >> > HDFS-13941 >> > >> > Console output >> > >> > https://builds.apache.org/job/PreCommit-HDFS-Build/25154/console >> > >> > Powered by >> > >> > Apache Yetus 0.8.0 http://yetus.apache.org >> > >> > >> > >> > -1 overall >> > >> > >> > Vote >> > >> > Subsystem >> > >> > Runtime >> > >> > Comment >> > >> > 0 >> > >> > reexec >> > >> > 0m 0s >> > >> > Docker mode activated. >> > >> > -1 >> > >> > patch >> > >> > 0m 4s >> > >> > HDFS-13877 does not apply to trunk. Rebase required? Wrong Branch? See >> > https://wiki.apache.org/hadoop/HowToContribute for help. >> > >> > >> > Subsystem >> > >> > Report/Notes >> > >> > JIRA Issue >> > >> > HDFS-13877 >> > >> > Console output >> > >> > https://builds.apache.org/job/PreCommit-HDFS-Build/25155/console >> > >> > Powered by >> > >> > Apache Yetus 0.8.0 http://yetus.apache.org >> > >> > >> > >> > >> > >> > Thanks, >> > Ajay Kumar >> > >> >
Re: Jenkins precommit build for HDFS failing
I tried to upgrade the yetus version for hbase pre commit job and the error message is the same with hadoop then. And the problem is that when fetching the patch we get a 4xx, so we just apply a nonexistent file... jira_http_fetch: https://issues.apache.org/jira/browse/HBASE-21233 returned 4xx status code. Maybe incorrect username/password? Ted Yu 于2018年9月28日周五 上午6:20写道： > Over in hbase precommit, I saw this: > https://builds.apache.org/job/PreCommit-HBASE-Build/14514/console > > Resolving deltas: 86% (114758/133146), completed with 87 local > objects.*21:06:47* fatal: pack has 18388 unresolved deltas*21:06:47* > fatal: index-pack failed*21:06:47* *21:06:47* at > > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)*21:06:47* > at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1721)*21:06:47* > at > org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72) > > > It seems QA machine(s) may have trouble accessing git. > > > I wonder if the 'index-pack failed' error would lead to patch not > being recognized. > > > FYI > > > On Thu, Sep 27, 2018 at 3:02 PM Ajay Kumar > wrote: > > > Hi, > > > > Jenkins precommit build for HDFS is failing with error that patch doesn’t > > apply to trunk, even when patch applies to trunk. > > I see other build failures with same error for other patches as well. > > Wanted to reach out to know if it is a known issue. > > > > > > Vote > > > > Subsystem > > > > Runtime > > > > Comment > > > > 0 > > > > reexec > > > > 0m 0s > > > > Docker mode activated. > > > > -1 > > > > patch > > > > 0m 5s > > > > HDFS-13941 does not apply to trunk. Rebase required? Wrong Branch? See > > https://wiki.apache.org/hadoop/HowToContribute for help. > > > > > > Subsystem > > > > Report/Notes > > > > JIRA Issue > > > > HDFS-13941 > > > > Console output > > > > https://builds.apache.org/job/PreCommit-HDFS-Build/25154/console > > > > Powered by > > > > Apache Yetus 0.8.0 http://yetus.apache.org > > > > > > > > -1 overall > > > > > > Vote > > > > Subsystem > > > > Runtime > > > > Comment > > > > 0 > > > > reexec > > > > 0m 0s > > > > Docker mode activated. > > > > -1 > > > > patch > > > > 0m 4s > > > > HDFS-13877 does not apply to trunk. Rebase required? Wrong Branch? See > > https://wiki.apache.org/hadoop/HowToContribute for help. > > > > > > Subsystem > > > > Report/Notes > > > > JIRA Issue > > > > HDFS-13877 > > > > Console output > > > > https://builds.apache.org/job/PreCommit-HDFS-Build/25155/console > > > > Powered by > > > > Apache Yetus 0.8.0 http://yetus.apache.org > > > > > > > > > > > > Thanks, > > Ajay Kumar > > >
[jira] [Created] (HDFS-13756) Implement the FileSystemLinkResolver in an async way
Duo Zhang created HDFS-13756: Summary: Implement the FileSystemLinkResolver in an async way Key: HDFS-13756 URL: https://issues.apache.org/jira/browse/HDFS-13756 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13755) Add retry support for async rpc
Duo Zhang created HDFS-13755: Summary: Add retry support for async rpc Key: HDFS-13755 URL: https://issues.apache.org/jira/browse/HDFS-13755 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13754) Add authentication support for async rpc
Duo Zhang created HDFS-13754: Summary: Add authentication support for async rpc Key: HDFS-13754 URL: https://issues.apache.org/jira/browse/HDFS-13754 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13753) Implement the mkdir API for AsyncDistributedFileSystem
Duo Zhang created HDFS-13753: Summary: Implement the mkdir API for AsyncDistributedFileSystem Key: HDFS-13753 URL: https://issues.apache.org/jira/browse/HDFS-13753 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang As it only need to connect NN, and easy to write a UT to verify it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13643) Implement basic async rpc client
Duo Zhang created HDFS-13643: Summary: Implement basic async rpc client Key: HDFS-13643 URL: https://issues.apache.org/jira/browse/HDFS-13643 Project: Hadoop HDFS Issue Type: Sub-task Components: ipc Reporter: Duo Zhang Implement the basic async rpc client so we can start working on the DFSClient implementation ASAP. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-13640) enable ShortCircuit Read on UC block
[ https://issues.apache.org/jira/browse/HDFS-13640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HDFS-13640: -- > enable ShortCircuit Read on UC block > > > Key: HDFS-13640 > URL: https://issues.apache.org/jira/browse/HDFS-13640 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.2 >Reporter: Gang Xie >Priority: Major > > The ShortCircuit read is disabled by HDFS-2757 due to the inconsistency of > the block states. With this limitation, some streaming/messaging application > could not benefit the performance improvement from SCR. In our streaming > system whose storage is HDFS, has around 90% read on the last blocks. So, > it's necessary to enable SCR on the last block, especially the app could > ensure the read after the flush. > After look into the original issue in HDFS-2757, it could only happen when > the read beyond the flush or the local datanode is kicked out from the > pipeline. But if there is the data and the visible length of the block covers > the read length, we could still read the data from the block, right? > > I didn't get a completed solution here. Any suggestion could be helpful. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13640) s
[ https://issues.apache.org/jira/browse/HDFS-13640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HDFS-13640. -- Resolution: Invalid Accident? > s > - > > Key: HDFS-13640 > URL: https://issues.apache.org/jira/browse/HDFS-13640 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.2 >Reporter: Gang Xie >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSSION] Create a branch to work on non-blocking access to HDFS
Will prepare a design doc soon to roughly describe the things we want to do and how we plan to do it, and also the undecided things, such as how to support fan-out. Thanks. 2018-05-04 4:54 GMT+08:00 Anu Engineer : > Hi St.ack/Wei-Chiu, > > It is very kind of St.Ack to bring this question to HDFS Dev. I think this > is a good feature to have. As for the branch question, > HDFS-9924 branch is already open, we could just use that and I am +1 on > adding Duo as a branch committer. > > I am not familiar with HBase code base, I am presuming that there will be > some deviation from the current design > doc posted in HDFS-9924. Would it be make sense to post a new design > proposal on HDFS-9924? > > --Anu > > > > On 5/3/18, 9:29 AM, "Wei-Chiu Chuang" wrote: > > Given that HBase 2 uses async output by default, the way that code is > maintained today in HBase is not sustainable. That piece of code > should be > maintained in HDFS. I am +1 as a participant in both communities. > > On Thu, May 3, 2018 at 9:14 AM, Stack wrote: > > > Ok with you lot if a few of us open a branch to work on a > non-blocking HDFS > > client? > > > > Intent is to finish up the old issue "HDFS-9924 [umbrella] > Nonblocking HDFS > > Access". On the foot of this umbrella JIRA is a proposal by the > > heavy-lifter, Duo Zhang. Over in HBase, we have a limited async DFS > client > > (written by Duo) that we use making Write-Ahead Logs. We call it > > AsyncFSWAL. It was shipped as the default WAL writer in hbase-2.0.0. > > > > Let me quote Duo from his proposal at the base of HDFS-9924: > > > > We use lots of internal APIs of HDFS to implement the > AsyncFSWAL, so it > > is expected that things like HBASE-20244 > > <https://issues.apache.org/jira/browse/HBASE-20244> > > ["NoSuchMethodException > > when retrieving private method decryptEncryptedDataEncryptionKey > from > > DFSClient"] will happen again and again. > > > > To make life easier, we need to move the async output related code > into > > HDFS. The POC [attached as patch on HDFS-9924] shows that option 3 > [1] can > > work, so I would like to create a feature branch to implement the > async dfs > > client. In general I think there are 4 steps: > > > > 1. Implement an async rpc client with option 3 [1] described above. > > 2. Implement the filesystem APIs which only need to connect to NN, > such as > > 'mkdirs'. > > 3. Implement async file read. The problem is the API. For pread I > think a > > CompletableFuture is enough, the problem is for the streaming read. > Need to > > discuss later. > > 4. Implement async file write. The API will also be a problem, but a > more > > important problem is that, if we want to support fan-out, the > current logic > > at DN side will make the semantic broken as we can read uncommitted > data > > very easily. In HBase it is solved by HBASE-14004 > > <https://issues.apache.org/jira/browse/HBASE-14004> but I do not > think we > > should keep the broken behavior in HDFS. We need to find a way to > deal with > > it. > > > > Comments welcome. > > > > Intent is to make a branch named HDFS-9924 (or should we just do a > new > > JIRA?) and to add Duo as a feature branch committer. If all goes > well, > > we'll call for a merge VOTE. > > > > Thanks, > > St.Ack > > > > 1.Option 3: "Use the old protobuf rpc interface and implement a new > rpc > > framework. The benefit is that we also do not need port unification > service > > at server side and do not need to maintain two implementations at > server > > side. And one more thing is that we do not need to upgrade protobuf > to > > 3.x." > > > > > > -- > A very happy Hadoop contributor > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
[jira] [Created] (HDFS-9372) Typo in DataStorage.recoverTransitionRead
Duo Zhang created HDFS-9372:
---
Summary: Typo in DataStorage.recoverTransitionRead
Key: HDFS-9372
URL: https://issues.apache.org/jira/browse/HDFS-9372
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Duo Zhang
{code:title=DataStorage.java}
if (this.initialized) {
LOG.info("DataNode version: " +
HdfsServerConstants.DATANODE_LAYOUT_VERSION
+ " and NameNode layout version: " + nsInfo.getLayoutVersion());
this.storageDirs = new ArrayList(dataDirs.size());
// mark DN storage is initialized
this.initialized = true;
}
{code}
The first if should be {{!this.initialized}} I think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8683) Implement flow control
Duo Zhang created HDFS-8683: --- Summary: Implement flow control Key: HDFS-8683 URL: https://issues.apache.org/jira/browse/HDFS-8683 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang We have implemented read block over http2 on the POC branch. https://github.com/Apache9/hadoop/tree/HDFS-7966-POC The block reader is implemented using jetty. We wrote a testcase to test performance with MiniCluster and we found that the the flow control of HTTP/2 has a big impact on the performance. Window update frame will be delayed if we create many threads to read, and netty will stop sending data if there is no window space left and thus cause a bad impact on performance. Flow control is a built-in feature of HTTP/2. We ignore it does not mean we can bypass it. So I think we need to support this feature at first place. Or at least, find a way to bypass it(maybe a very large initial window size?) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8671) Add client support for http2 stream channels
Duo Zhang created HDFS-8671:
---
Summary: Add client support for http2 stream channels
Key: HDFS-8671
URL: https://issues.apache.org/jira/browse/HDFS-8671
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Duo Zhang
{{Http2StreamChannel}} is introduced in HDFS-8515 but can only be used at
server side.
Now we implement Http2BlockReader using jetty http2-client in the POC branch,
but the final version of jetty 9.3.0 only accepts java8.
So here we plan to extend the functions of {{Http2StreamChannel}} to support
client side usage and then implement Http2BlockReader based on it. And we still
use jetty http2-client to write testcases to ensure that our http2
implementation is valid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8515) Abstract a DTP/2 HTTP/2 server
Duo Zhang created HDFS-8515: --- Summary: Abstract a DTP/2 HTTP/2 server Key: HDFS-8515 URL: https://issues.apache.org/jira/browse/HDFS-8515 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang Assignee: Duo Zhang Discussed in HDFS-8471. https://issues.apache.org/jira/browse/HDFS-8471?focusedCommentId=14568196&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14568196 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8472) Implement a new block reader that reads data over HTTP/2
Duo Zhang created HDFS-8472: --- Summary: Implement a new block reader that reads data over HTTP/2 Key: HDFS-8472 URL: https://issues.apache.org/jira/browse/HDFS-8472 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8471) Implement read block over HTTP/2
Duo Zhang created HDFS-8471: --- Summary: Implement read block over HTTP/2 Key: HDFS-8471 URL: https://issues.apache.org/jira/browse/HDFS-8471 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Duo Zhang Assignee: Duo Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
