Re: [DISCUSS] hadoop branch-3.3+ going to java11 only
In theory, I like the idea of setting aside Java 8. Unfortunately, I don't know that upgrading within the 3.3 line adheres to our binary compatibility policy [1]. I don't see specific discussion of the Java version there, but it states that you should be able to drop in minor upgrades and have existing apps keep working. Users might find it surprising if they try to upgrade a cluster that has JDK 8. There is also the question of impact on downstream projects [2]. We'd have to check plans with our consumers. What about the idea of shooting for a 3.4 release on JDK 11 (or even 17)? The downside is that we'd probably need to set boundaries on end of life/limited support for 3.2 and 3.3 to keep the workload manageable. [1] https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html#Java_Binary_compatibility_for_end-user_applications_i.e._Apache_Hadoop_ABI [2] https://github.com/apache/spark/blob/v3.3.2/pom.xml#L109 Chris Nauroth On Tue, Mar 28, 2023 at 11:10 AM Ayush Saxena wrote: > > > > it's already hard to migrate from JDK8 why not retarget JDK17. > > > > +1, makes sense to me, sounds like a win-win situation to me, though there > would be some additional issues to chase now :) > > -Ayush > > > On Tue, 28 Mar 2023 at 23:29, Wei-Chiu Chuang wrote: > > > My random thoughts. Probably bad takes: > > > > There are projects experimenting with JDK17 now. > > JDK11 active support will end in 6 months. If it's already hard to > migrate > > from JDK8 why not retarget JDK17. > > > > On Tue, Mar 28, 2023 at 10:30 AM Ayush Saxena > wrote: > > > >> I know Jersey upgrade as a blocker. Some folks were chasing that last > >> year during 3.3.4 time, I don’t know where it is now, didn’t see then > >> what’s the problem there but I remember there was some intitial PR which > >> did it for HDFS atleast, so I never looked beyond that… > >> > >> I too had jdk-11 in my mind, but only for trunk. 3.4.x can stay as > >> java-11 only branch may be, but that is something later to decide, once > we > >> get the code sorted… > >> > >> -Ayush > >> > >> > On 28-Mar-2023, at 9:16 PM, Steve Loughran > > >> wrote: > >> > > >> > well, how about we flip the switch and get on with it. > >> > > >> > slf4j seems happy on java11, > >> > > >> > side issue, anyone seen test failures on zulu1.8; somehow my test run > is > >> > failing and i'm trying to work out whether its a mismatch in command > >> > line/ide jvm versions, or the 3.3.5 JARs have been built with an > openjdk > >> > version which requires IntBuffer implements an overridden method > >> IntBuffer > >> > rewind(). > >> > > >> > java.lang.NoSuchMethodError: > >> java.nio.IntBuffer.rewind()Ljava/nio/IntBuffer; > >> > > >> > at > >> org.apache.hadoop.fs.FSInputChecker.verifySums(FSInputChecker.java:341) > >> > at > >> > > >> > org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:308) > >> > at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:257) > >> > at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:202) > >> > at java.io.DataInputStream.read(DataInputStream.java:149) > >> > > >> >> On Tue, 28 Mar 2023 at 15:52, Viraj Jasani > wrote: > >> >> IIRC some of the ongoing major dependency upgrades (log4j 1 to 2, > >> jersey 1 > >> >> to 2 and junit 4 to 5) are blockers for java 11 compile + test > >> stability. > >> >> On Tue, Mar 28, 2023 at 4:55 AM Steve Loughran > >> >> >> wrote: > >> >>> Now that hadoop 3.3.5 is out, i want to propose something new > >> >>> we switch branch-3.3 and trunk to being java11 only > >> >>> 1. java 11 has been out for years > >> >>> 2. oracle java 8 is no longer available under "premier support"; you > >> >>> can't really get upgrades > >> >>> > https://www.oracle.com/java/technologies/java-se-support-roadmap.html > >> >>> 3. openJDK 8 releases != oracle ones, and things you compile with > them > >> >>> don't always link to oracle java 8 (some classes in java.nio have > >> >> added > >> >>> more overrides) > >> >>> 4. more and more libraries we want to upgrade to/bundle are java 11 > >> >> only > >> >>> 5. moving to java 11 would cut our yetus build workload in half, and > >> >>> line up for adding java 17 builds instead. > >> >>> I know there are some outstanding issues still in > >> >>> https://issues.apache.org/jira/browse/HADOOP-16795 -but are they > >> >> blockers? > >> >>> Could we just move to java11 and enhance at our leisure, once java8 > >> is no > >> >>> longer a concern. > >> > >> - > >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org > >> > >> >
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
+1 Thank you for the release candidate, Steve! * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests passed. * mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 * Checked dependency tree to make sure we have all of the expected library updates that are mentioned in the release notes. * mvn -o dependency:tree * Confirmed that hadoop-openstack is now just a stub placeholder artifact with no code. * For ARM verification: * Ran "file " on all native binaries in the ARM tarball to confirm they actually came out with ARM as the architecture. * Output of hadoop checknative -a on ARM looks good. * Ran a MapReduce job with the native bzip2 codec for compression, and it worked fine. * Ran a MapReduce job with YARN configured to use LinuxContainerExecutor and verified launching the containers through container-executor worked. Chris Nauroth On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena wrote: > +1(Binding) > > * Built from source (x86 & ARM) > * Successful Native Build (x86 & ARM) > * Verified Checksums (x86 & ARM) > * Verified Signature (x86 & ARM) > * Checked the output of hadoop version (x86 & ARM) > * Verified the output of hadoop checknative (x86 & ARM) > * Ran some basic HDFS shell commands. > * Ran some basic Yarn shell commands. > * Played a bit with HDFS Erasure Coding. > * Ran TeraGen & TeraSort > * Browed through NN, DN, RM & NM UI > * Skimmed over the contents of website. > * Skimmed over the contents of maven repo. > * Selectively ran some HDFS & CloudStore tests > > Thanx Steve for driving the release. Good Luck!!! > > -Ayush > > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He wrote: > > > > +1 > > > > * Verified signature and checksum of the source tarball. > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package > > -DskipTests -Pnative -Pdist -Dtar`. > > * Setup pseudo cluster with HDFS and YARN. > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the > > result. > > * Run example mr applications and check the result - Pi & wordcount. > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc. > > > > Thanks Steve for your work. > > > > Best Regards, > > - He Xiaoqiao > > > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki < > iwasak...@oss.nttdata.com> > >> wrote: > >> > >> +1 > >> > >> + verified the signature and checksum of the source tarball. > >> > >> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8 > >> with native profile enabled. > >> + launched pseudo distributed cluster including kms and httpfs with > >> Kerberos and SSL enabled. > >> + created encryption zone, put and read files via httpfs. > >> + ran example MR wordcount over encryption zone. > >> + checked the binary of container-executor. > >> > >> + built rpm packages by Bigtop (with trivial modifications) on Rocky > Linux > >> 8 (aarch64). > >> + ran smoke-tests of hdfs, yarn and mapreduce. > >> + built site documentation and skimmed the contents. > >> + Javadocs are contained. > >> > >> Thanks, > >> Masatake Iwasaki > >> > >>> On 2023/03/16 4:47, Steve Loughran wrote: > >>> Apache Hadoop 3.3.5 > >>> > >>> Mukund and I have put together a release candidate (RC3) for Hadoop > >> 3.3.5. > >>> > >>> What we would like is for anyone who can to verify the tarballs, > >> especially > >>> anyone who can try the arm64 binaries as we want to include them too. > >>> > >>> The RC is available at: > >>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ > >>> > >>> The git tag is release-3.3.5-RC3, commit 706d88266ab > >>> > >>> The maven artifacts are staged at > >>> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/ > >>> > >>> You can find my public key at: > >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > >>> > >>> Change log > >>> > >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md > >>> > >>> Release notes > >>>
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
Yes, I'm in progress on verification, so you can expect to get a vote from me. Thank you, Steve! Chris Nauroth On Sat, Mar 18, 2023 at 9:19 AM Ashutosh Gupta wrote: > Hi Steve > > I will also do it by today/tomorrow. > > Thanks, > Ashutosh > > On Sat, 18 Mar, 2023, 4:07 pm Steve Loughran, > > wrote: > > > Thank you for this! > > > > Can anyone else with time do a review too? i really want to get this one > > done, now the HDFS issues are all resolved. > > > > I do not want this release to fall by the wayside through lack of votes > > alone. In fact, I would be very unhappy > > > > > > > > On Sat, 18 Mar 2023 at 06:47, Viraj Jasani wrote: > > > > > +1 (non-binding) > > > > > > * Signature/Checksum: ok > > > * Rat check (1.8.0_341): ok > > > - mvn clean apache-rat:check > > > * Built from source (1.8.0_341): ok > > > - mvn clean install -DskipTests > > > * Built tar from source (1.8.0_341): ok > > > - mvn clean package -Pdist -DskipTests -Dtar > -Dmaven.javadoc.skip=true > > > > > > Containerized deployments: > > > * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper > 3.7 > > > * Deployed and started JHS, RM, NM > > > * Hbase, hdfs CRUD looks good > > > * Sample RowCount MapReduce job looks good > > > > > > * S3A tests with scale profile looks good > > > > > > > > > On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran > > > > > > wrote: > > > > > > > Apache Hadoop 3.3.5 > > > > > > > > Mukund and I have put together a release candidate (RC3) for Hadoop > > > 3.3.5. > > > > > > > > What we would like is for anyone who can to verify the tarballs, > > > especially > > > > anyone who can try the arm64 binaries as we want to include them too. > > > > > > > > The RC is available at: > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ > > > > > > > > The git tag is release-3.3.5-RC3, commit 706d88266ab > > > > > > > > The maven artifacts are staged at > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1369/ > > > > > > > > You can find my public key at: > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > Change log > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md > > > > > > > > Release notes > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md > > > > > > > > This is off branch-3.3 and is the first big release since 3.3.2. > > > > > > > > Key changes include > > > > > > > > * Big update of dependencies to try and keep those reports of > > > > transitive CVEs under control -both genuine and false positives. > > > > * HDFS RBF enhancements > > > > * Critical fix to ABFS input stream prefetching for correct reading. > > > > * Vectored IO API for all FSDataInputStream implementations, with > > > > high-performance versions for file:// and s3a:// filesystems. > > > > file:// through java native io > > > > s3a:// parallel GET requests. > > > > * This release includes Arm64 binaries. Please can anyone with > > > > compatible systems validate these. > > > > * and compared to the previous RC, all the major changes are > > > > HDFS issues. > > > > > > > > Note, because the arm64 binaries are built separately on a different > > > > platform and JVM, their jar files may not match those of the x86 > > > > release -and therefore the maven artifacts. I don't think this is > > > > an issue (the ASF actually releases source tarballs, the binaries are > > > > there for help only, though with the maven repo that's a bit > blurred). > > > > > > > > The only way to be consistent would actually untar the x86.tar.gz, > > > > overwrite its binaries with the arm stuff, retar, sign and push out > > > > for the vote. Even automating that would be risky. > > > > > > > > Please try the release and vote. The vote will run for 5 days. > > > > > > > > -Steve > > > > > > > > > >
[jira] [Resolved] (HDFS-16891) Avoid the overhead of copy-on-write exception list while loading inodes sub sections in parallel
[ https://issues.apache.org/jira/browse/HDFS-16891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-16891. -- Fix Version/s: 3.4.0 3.3.9 Resolution: Fixed > Avoid the overhead of copy-on-write exception list while loading inodes sub > sections in parallel > > > Key: HDFS-16891 > URL: https://issues.apache.org/jira/browse/HDFS-16891 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.4 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > If we enable parallel loading and persisting of inodes from/to fs image, we > get the benefit of improved performance. However, while loading sub-sections > INODE_DIR_SUB and INODE_SUB, if we encounter any errors, we use copy-on-write > list to maintain the list of exceptions. Since our usecase is not to iterate > over this list while executor threads are adding new elements to the list, > using copy-on-write is bit of an overhead for this usecase. > It would be better to synchronize adding new elements to the list rather than > having the list copy all elements over every time new element is added to the > list. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16887) Log start and end of phase/step in startup progress
[ https://issues.apache.org/jira/browse/HDFS-16887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-16887. -- Fix Version/s: 3.4.0 3.2.5 3.3.9 Resolution: Fixed > Log start and end of phase/step in startup progress > --- > > Key: HDFS-16887 > URL: https://issues.apache.org/jira/browse/HDFS-16887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.5, 3.3.9 > > > As part of Namenode startup progress, we have multiple phases and steps > within phase that are instantiated. While the startup progress view can be > instantiated with the current view of phase/step, having at least DEBUG logs > for startup progress would be helpful to identify when a particular step for > LOADING_FSIMAGE/SAVING_CHECKPOINT/LOADING_EDITS was started and ended. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5
Is it a problem limited to MiniDFSCluster, or is it a broader problem of RPC client resource cleanup? The patch is changing connection close cleanup, so I assumed the latter. If so, then it could potentially impact applications integrating with the RPC clients. If the problem is limited to MiniDFSCluster and restarts within a single JVM, then I agree the impact is smaller. Then, we'd want to consider what downstream projects have tests that do restarts on a MiniDFSCluster. Chris Nauroth On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena wrote: > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it >> mentioned anywhere in change log or release notes. Are they actually >> up-to-date? > > > I don't think there is any issue with the ReleaseNotes generation as such > but with the Resolution type of this ticket, It ain't marked as Fixed but > Done. The other ticket which is marked Done is also not part of the release > notes. [1] > > if I'm understanding the potential impact of HDFS-16853 >> correctly, then it's serious enough to fix before a release. (I could >> change my vote if someone wants to make a case that it's not that >> serious.) >> > > Chris, I just had a very quick look at HDFS-16853, I am not sure if this > can happen outside a MiniDfsCluster setup? Just guessing from the > description in the ticket. It looked like when we did a restart of the > Namenode in the MiniDfsCluster, I guess that would be in the same single > JVM, and that is why a previous blocked thread caused issues with the > restart. That is what I understood, I haven't checked the code though. > > Second, In the same context, Being curious If this lands up being a > MiniDfsCluster only issue, do we still consider this a release blocker? Not > saying in a way it won't be serious, MiniDfsCluster is very widely used by > downstream projects and all, so just wanted to know > > Regarding the Hive & Bouncy castle. The PR seems to have a valid binding > veto, I am not sure if it will get done any time soon, so if the use case > is something required, I would suggest handling it at Hadoop itself. It > seems to be centric to Hive-3.x, I tried compiling the Hive master branch > with 3.3.5 and it passed. Other than that Hive officially support only > Hadoop-3.3.1 and that too only in the last 4.x release[2] > > > [1] > https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC > [2] https://issues.apache.org/jira/browse/HIVE-24484 > > -Ayush > > On Tue, 3 Jan 2023 at 23:51, Chris Nauroth wrote: > >> -1, because if I'm understanding the potential impact of HDFS-16853 >> correctly, then it's serious enough to fix before a release. (I could >> change my vote if someone wants to make a case that it's not that >> serious.) >> >> Otherwise, this RC was looking good: >> >> * Verified all checksums. >> * Verified all signatures. >> * Built from source, including native code on Linux. >> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy >> -Drequire.zstd -DskipTests >> * Tests passed. >> * mvn --fail-never clean test -Pnative -Dparallel-tests >> -Drequire.snappy -Drequire.zstd -Drequire.openssl >> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 >> * Checked dependency tree to make sure we have all of the expected library >> updates that are mentioned in the release notes. >> * mvn -o dependency:tree >> * Farewell, S3Guard. >> * Confirmed that hadoop-openstack is now just a stub placeholder artifact >> with no code. >> * For ARM verification: >> * Ran "file " on all native binaries in the ARM tarball to confirm >> they actually came out with ARM as the architecture. >> * Output of hadoop checknative -a on ARM looks good. >> * Ran a MapReduce job with the native bzip2 codec for compression, and >> it worked fine. >> * Ran a MapReduce job with YARN configured to use >> LinuxContainerExecutor and verified launching the containers through >> container-executor worked. >> >> My local setup didn't have the test failures mentioned by Viraj, though >> there was some flakiness with a few HDFS snapshot tests timing out. >> >> Regarding Hive and Bouncy Castle, there is an existing issue and pull >> request tracking an upgrade attempt. It's looking like some amount of code >> changes are required: >> >> https://issues.apache.org/jira/browse/HIVE-26648 >> https://github.com/apache/hive/pull/3744
Re: [VOTE] Release Apache Hadoop 3.3.5
-1, because if I'm understanding the potential impact of HDFS-16853 correctly, then it's serious enough to fix before a release. (I could change my vote if someone wants to make a case that it's not that serious.) Otherwise, this RC was looking good: * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests passed. * mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 * Checked dependency tree to make sure we have all of the expected library updates that are mentioned in the release notes. * mvn -o dependency:tree * Farewell, S3Guard. * Confirmed that hadoop-openstack is now just a stub placeholder artifact with no code. * For ARM verification: * Ran "file " on all native binaries in the ARM tarball to confirm they actually came out with ARM as the architecture. * Output of hadoop checknative -a on ARM looks good. * Ran a MapReduce job with the native bzip2 codec for compression, and it worked fine. * Ran a MapReduce job with YARN configured to use LinuxContainerExecutor and verified launching the containers through container-executor worked. My local setup didn't have the test failures mentioned by Viraj, though there was some flakiness with a few HDFS snapshot tests timing out. Regarding Hive and Bouncy Castle, there is an existing issue and pull request tracking an upgrade attempt. It's looking like some amount of code changes are required: https://issues.apache.org/jira/browse/HIVE-26648 https://github.com/apache/hive/pull/3744 Chris Nauroth On Tue, Jan 3, 2023 at 8:57 AM Chao Sun wrote: > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it > mentioned anywhere in change log or release notes. Are they actually > up-to-date? > > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki > wrote: > > > > >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to > dependency change. > > > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not > expected in hbase-shaded-with-hadoop-check-invariants. > > Updating hbase-shaded/pom.xml is expected to be the fix as done in > HBASE-27292. > > > https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277 > > > >[INFO] --- exec-maven-plugin:1.6.0:exec > (check-jar-contents-for-stuff-with-hadoop) @ > hbase-shaded-with-hadoop-check-invariants --- > >[ERROR] Found artifact with unexpected contents: > '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar' > >Please check the following and either correct the build or update > >the allowed list with reasoning. > > > >com/ > >com/sun/ > >com/sun/jersey/ > >com/sun/jersey/json/ > >... > > > > > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem > to be problematic. > > Excluding them on hive-jdbc might be the fix. > > > >[ERROR] Failed to execute goal > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on > project hive-jdbc: Error creating shaded jar: Problem shading JAR > /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar > entry > META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class: > java.lang.IllegalArgumentException: Unsupported class file major version 59 > -> [Help 1] > >... > > > > > > On 2023/01/02 22:02, Masatake Iwasaki wrote: > > > Thanks for your great effort for the new release, Steve and Mukund. > > > > > > +1 while it would be nice if we can address missed Javadocs. > > > > > > + verified the signature and checksum. > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native > profile enabled. > > >+ launched pseudo distributed cluster including kms and httpfs with > Kerberos and SSL enabled. > > >+ created encryption zone, put and read files via httpfs. > > >+ ran example MR wordcount over encryption zone. > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 > (both x86_64 and aarch64). > > >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to > dependency change. > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 > worked. > > > + skimmed the site contents. > > >-
Re: [VOTE] Release Apache Hadoop 3.3.5
I'm not quite ready to vote yet, pending some additional testing. However, I wanted to give a quick update that ARM support is looking good from my perspective. I focused on verifying the native bits that would need to be different for ARM vs. x64. Here is what I did: * Ran "file " on all native binaries in the ARM tarball to confirm they actually came out with ARM as the architecture. * Output of hadoop checknative -a on ARM looks good. * Ran a MapReduce job with the native bzip2 codec for compression, and it worked fine. * Ran a MapReduce job with YARN configured to use LinuxContainerExecutor and verified launching the containers through container-executor worked. Chris Nauroth On Wed, Dec 21, 2022 at 11:29 AM Steve Loughran wrote: > Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5. > > Given the time of year it's a bit unrealistic to run a 5 day vote and > expect people to be able to test it thoroughly enough to make this the one > we can ship. > > What we would like is for anyone who can to verify the tarballs, and test > the binaries, especially anyone who can try the arm64 binaries. We've got > the building of those done and now the build file will incorporate them > into the release -but neither of us have actually tested it yet. Maybe I > should try it on my pi400 over xmas. > > The maven artifacts are up on the apache staging repo -they are the ones > from x86 build. Building and testing downstream apps will be incredibly > helpful. > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/ > > The git tag is release-3.3.5-RC0, commit 3262495904d > > The maven artifacts are staged at > https://repository.apache.org/content/repositories/orgapachehadoop-1365/ > > You can find my public key at: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > Change log > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md > > Release notes > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md > > This is off branch-3.3 and is the first big release since 3.3.2. > > Key changes include > > * Big update of dependencies to try and keep those reports of > transitive CVEs under control -both genuine and false positive. > * HDFS RBF enhancements > * Critical fix to ABFS input stream prefetching for correct reading. > * Vectored IO API for all FSDataInputStream implementations, with > high-performance versions for file:// and s3a:// filesystems. > file:// through java native io > s3a:// parallel GET requests. > * This release includes Arm64 binaries. Please can anyone with > compatible systems validate these. > > > Please try the release and vote on it, even though i don't know what is a > good timeline here...i'm actually going on holiday in early jan. Mukund is > around and so can drive the process while I'm offline. > > Assuming we do have another iteration, the RC1 will not be before mid jan > for that reason > > Steve (and mukund) >
[jira] [Resolved] (HDFS-8510) Provide different timeout settings for hdfs dfsadmin -getDatanodeInfo.
[
https://issues.apache.org/jira/browse/HDFS-8510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-8510.
-
Resolution: Won't Fix
This is an old improvement proposal that I'm no longer planning on
implementing. I'm going to close the issue. If anyone else would find it
useful, please feel free to reopen and reassign. I'd be happy to help with code
review.
> Provide different timeout settings for hdfs dfsadmin -getDatanodeInfo.
> --
>
> Key: HDFS-8510
> URL: https://issues.apache.org/jira/browse/HDFS-8510
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
>
> During a rolling upgrade, an administrator runs {{hdfs dfsadmin
> -getDatanodeInfo}} to check if a DataNode has stopped. Currently, this
> operation is subject to the RPC connection retries defined in
> {{ipc.client.connect.max.retries}} and {{ipc.client.connect.retry.interval}}.
> This issue proposes adding separate configuration properties to control the
> retries for this operation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-4289) FsDatasetImpl#updateReplicaUnderRecovery throws errors validating replica byte count on Windows
[
https://issues.apache.org/jira/browse/HDFS-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-4289.
-
Resolution: Won't Fix
I'm no longer actively working on this. I no longer have easy access to a
Windows environment to make Windows-specific changes, or even to confirm that
this test failure still happens. It's a very old issue with no recent activity,
so I'm going to assume it's no longer relevant and close it out. If it's still
an ongoing issue that a Windows developer wants to pick up, please feel free to
reopen and reassign.
> FsDatasetImpl#updateReplicaUnderRecovery throws errors validating replica
> byte count on Windows
> ---
>
> Key: HDFS-4289
> URL: https://issues.apache.org/jira/browse/HDFS-4289
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: trunk-win
> Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
>
> {{FsDatasetImpl#updateReplicaUnderRecovery}} throws errors validating replica
> byte count on Windows. This can be seen by running
> {{TestBalancerWithNodeGroup#testBalancerWithRackLocality}}, which fails on
> Windows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.4
+1 (binding) * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests passed. * mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 * Checked dependency tree to make sure we have all of the expected library updates that are mentioned in the release notes. * mvn -o dependency:tree I saw a LibHDFS test failure, but I know it's something flaky that's already tracked in a JIRA issue. The release looks good. Steve, thank you for driving this. Chris Nauroth On Wed, Aug 3, 2022 at 11:27 AM Steve Loughran wrote: > my vote for this is +1, binding. > > obviously I`m biased, but i do not want to have to issue any more interim > releases before the feature release off branch-3.3, so I am trying to be > ruthless. > > my client vaidator ant project has a more targets to help with releasing, > and now builds a lot mor of my local projects > https://github.com/steveloughran/validate-hadoop-client-artifacts > all good as far as my test coverage goes, with these projects validating > the staged dependencies. > > now, who else can review > > On Fri, 29 Jul 2022 at 19:47, Steve Loughran wrote: > > > > > > > I have put together a release candidate (RC1) for Hadoop 3.3.4 > > > > The RC is available at: > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/ > > > > The git tag is release-3.3.4-RC1, commit a585a73c3e0 > > > > The maven artifacts are staged at > > https://repository.apache.org/content/repositories/orgapachehadoop-1358/ > > > > You can find my public key at: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > Change log > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md > > > > Release notes > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md > > > > There's a very small number of changes, primarily critical code/packaging > > issues and security fixes. > > > > See the release notes for details. > > > > Please try the release and vote. The vote will run for 5 days. > > > > steve > > >
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
I'm changing my vote to +1 (binding). Masatake and Ashutosh, thank you for investigating. I reran tests without the parallel options, and that mostly addressed the failures. Maybe the tests in question are just not sufficiently isolated to support parallel execution. That looks to be the case for TestFsck, where the failure was caused by missing audit log entries. This test works by toggling global logging state, so I can see why multi-threaded execution might confuse the test. Chris Nauroth On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta wrote: > +1(non-binding) > > * Builds from source look good. > * Checksums and signatures are correct. > * Running basic HDFS and MapReduce commands looks good. > > > * TestAMRMProxy - Not able to reproduce in local > > * TestFsck - I can see failure only I can see is > TestFsck.testFsckListCorruptSnapshotFiles which passed after applying > HDFS-15038 > > * TestSLSStreamAMSynth - Not able to reproduce in local > > * TestServiceAM - Not able to reproduce in local > > Thanks Masatake for driving this release. > > On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki < > iwasak...@oss.nttdata.com> > wrote: > > > Hi developers, > > > > I'm still waiting for your vote. > > I'm considering the intermittent test failures mentioned by Chris are not > > blocker. > > Please file a JIRA and let me know if you find a blocker issue. > > > > I will appreciate your help for the release process. > > > > Regards, > > Masatake Iwasaki > > > > On 2022/07/20 14:50, Masatake Iwasaki wrote: > > >> TestServiceAM > > > > > > I can see the reported failure of TestServiceAM in some "Apache Hadoop > > qbt Report: branch-3.2+JDK8 on Linux/x86_64". > > > 3.3.0 and above might be fixed by YARN-8867 which added guard using > > GenericTestUtils#waitFor for stabilizing the > > testContainersReleasedWhenPreLaunchFails. > > > YARN 8867 did not modified other code under hadoop-yarn-services. > > > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2. > > > > > > > > > On 2022/07/20 14:21, Masatake Iwasaki wrote: > > >> Thanks for testing the RC0, Chris. > > >> > > >>> The following are new test failures for me on 3.2.4: > > >>> * TestAMRMProxy > > >>> * TestFsck > > >>> * TestSLSStreamAMSynth > > >>> * TestServiceAM > > >> > > >> I could not reproduce the test failures on my local. > > >> > > >> For TestFsck, if the failed test case is > > testFsckListCorruptSnapshotFiles, > > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix. > > >> > > >> The failure of TestSLSStreamAMSynth looks frequently reported by > > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". > > >> It could be tagged as known flaky test. > > >> > > >> On 2022/07/20 9:15, Chris Nauroth wrote: > > >>> -0 (binding) > > >>> > > >>> * Verified all checksums. > > >>> * Verified all signatures. > > >>> * Built from source, including native code on Linux. > > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl > > -Drequire.snappy > > >>> -Drequire.zstd -DskipTests > > >>> * Tests mostly passed, but see below. > > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests > > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl > > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > > >>> > > >>> The following are new test failures for me on 3.2.4: > > >>> * TestAMRMProxy > > >>> * TestFsck > > >>> * TestSLSStreamAMSynth > > >>> * TestServiceAM > > >>> > > >>> The following tests also failed, but they also fail for me on 3.2.3, > so > > >>> they aren't likely to be related to this release candidate: > > >>> * TestCapacitySchedulerNodeLabelUpdate > > >>> * TestFrameworkUploader > > >>> * TestSLSGenericSynth > > >>> * TestSLSRunner > > >>> * test_libhdfs_threaded_hdfspp_test_shim_static > > >>> > > >>> I'm not voting a full -1, because I haven't done any root cause > > analysis on > > >>> these new test failures. I don't know if it's a quirk to my > > environment, > > >>> though I&
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
-0 (binding) * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests mostly passed, but see below. * mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM The following tests also failed, but they also fail for me on 3.2.3, so they aren't likely to be related to this release candidate: * TestCapacitySchedulerNodeLabelUpdate * TestFrameworkUploader * TestSLSGenericSynth * TestSLSRunner * test_libhdfs_threaded_hdfspp_test_shim_static I'm not voting a full -1, because I haven't done any root cause analysis on these new test failures. I don't know if it's a quirk to my environment, though I'm using the start-build-env.sh Docker container, so any build dependencies should be consistent. I'd be comfortable moving ahead if others are seeing these tests pass. Chris Nauroth On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki wrote: > +1 from myself. > > * skimmed the contents of site documentation. > > * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with > `-Pnative`. > > * launched pseudo distributed cluster including kms and httpfs with > Kerberos and SSL enabled. > >* created encryption zone, put and read files via httpfs. >* ran example MR wordcount over encryption zone. > > * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some > example MR jobs. > > * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0 >on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests. >https://github.com/apache/bigtop/pull/942 > >* Hive needs updating exclusion rule to address HADOOP-18088 (migration > to reload4j). > > * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository:: > > > staged > staged-releases > > https://repository.apache.org/content/repositories/orgapachehadoop-1354 > > > true > > > true > > > > Thanks, > Masatake Iwasaki > > On 2022/07/13 1:14, Masatake Iwasaki wrote: > > Hi all, > > > > Here's Hadoop 3.2.4 release candidate #0: > > > > The RC is available at: > >https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/ > > > > The RC tag is at: > >https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0 > > > > The Maven artifacts are staged at: > > > https://repository.apache.org/content/repositories/orgapachehadoop-1354 > > > > You can find my public key at: > >https://downloads.apache.org/hadoop/common/KEYS > > > > Please evaluate the RC and vote. > > The vote will be open for (at least) 5 days. > > > > Thanks, > > Masatake Iwasaki > > > > - > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
Re: HDFS-16577 - Request for input - Let administrator override connection details when registering datanodes
Hello Lars, I can't say I've personally run HDFS on Kubernetes with Kerberos enabled. However, some of the issues you raise sound like they have some overlap with the HDFS multi-homing features: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html Have you seen this? Does anything look helpful there? Chris Nauroth On Fri, Jun 24, 2022 at 4:55 AM Lars Francke wrote: > Hi everyone, > > we're trying to get HDFS running in Kubernetes using Kerberos. > This has some challenges as you might expect. > We have created an issue for that including a spike: > https://issues.apache.org/jira/browse/HDFS-16577 > > Currently (as of 3.2.2, but reading through the release notes this doesn't > seem to have changed since then) DataNodes use the same properties for > deciding which port to bind each service to, as for deciding which ports > are included in the `DatanodeRegistration` sent to the NameNode. Further, > NameNodes overwrite the DataNode's IP address with the incoming address > during registration. > > Both of these prevent external users from connecting to DataNodes that are > hosted behind some sort of NAT (such as Kubernetes). > > We'd go ahead with a proper implementation/PR but we thought about asking > for comments/feedback first. Maybe someone else has already done some work > here that we might have missed etc. > > Thank you! > > Cheers, > Lars >
[jira] [Resolved] (HDFS-16623) IllegalArgumentException in LifelineSender
[
https://issues.apache.org/jira/browse/HDFS-16623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-16623.
--
Hadoop Flags: Reviewed
Resolution: Fixed
I have committed this to trunk, branch-3.3 and branch-3.2. [~xuzq_zander] ,
thank you for the contribution.
> IllegalArgumentException in LifelineSender
> --
>
> Key: HDFS-16623
> URL: https://issues.apache.org/jira/browse/HDFS-16623
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In our production environment, an IllegalArgumentException occurred in the
> LifelineSender at one DataNode which was undergoing GC at that time.
> And the bug code is at line 1060 in BPServiceActor.java, because the sleep
> time is negative.
> {code:java}
> while (shouldRun()) {
> try {
> if (lifelineNamenode == null) {
> lifelineNamenode = dn.connectToLifelineNN(lifelineNnAddr);
> }
> sendLifelineIfDue();
> Thread.sleep(scheduler.getLifelineWaitTime());
> } catch (InterruptedException e) {
> Thread.currentThread().interrupt();
> } catch (IOException e) {
> LOG.warn("IOException in LifelineSender for " + BPServiceActor.this,
> e);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.10.2 - RC0
+1 (binding) * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Almost all unit tests passed. * mvn clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 * TestBookKeeperHACheckpoints consistently has a few failures. * TestCapacitySchedulerNodeLabelUpdate is flaky, intermittently timing out. These test failures don't look significant enough to hold up a release, so I'm still voting +1. Chris Nauroth On Sun, May 29, 2022 at 2:35 AM Masatake Iwasaki < iwasak...@oss.nttdata.co.jp> wrote: > Thanks for the help, Ayush. > > I committed HADOOP-16663/HADOOP-16664 and cherry-picked HADOOP-16985 to > branch-2.10 (and branch-3.2). > If I need to cut RC1, I will try cherry-picking them to branch-2.10.2 > > Masatake Iwasaki > > > On 2022/05/28 5:23, Ayush Saxena wrote: > > The checksum stuff was addressed in HADOOP-16985, so that filename stuff > is > > sorted only post 3.3.x > > BTW it is a known issue: > > > https://issues.apache.org/jira/browse/HADOOP-16494?focusedCommentId=16927236&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16927236 > > > > Must not be a blocker for us > > > > The RAT check failing with dependency issue. That also should work post > > 3.3.x because there is no Hadoop-maven-plugin dependency in > Hadoop-yarn-api > > module post 3.3.x, HADOOP-16560 removed it. > > Ref: > > > https://github.com/apache/hadoop/pull/1496/files#diff-f5d219eaf211871f9527ae48da59586e7e9958ea7649de74a1393e599caa6dd6L121-R122 > > > > So, that is why the RAT check passes for 3.3.x+ without the need of this > > module. Committing HADOOP-16663, should solve this though.(I haven't > tried > > though, just by looking at the problem) > > > > Good to have patches, but doesn't look like blockers to me. kind of build > > related stuffs only, nothing bad with our core Hadoop code. > > > > -Ayush > > > > On Sat, 28 May 2022 at 01:04, Viraj Jasani wrote: > > > >> +0 (non-binding), > >> > >> * Signature/Checksum looks good, though I am not sure where > >> "target/artifacts" is coming from for the tars, here is the diff (this > was > >> the case for 2.10.1 as well but checksum was correct): > >> > >> 1c1 > >> < SHA512 (hadoop-2.10.2-site.tar.gz) = > >> > >> > 3055a830003f5012660d92da68a317e15da5b73301c2c73cf618e724c67b7d830551b16928e0c28c10b66f04567e4b6f0b564647015bacc4677e232c0011537f > >> --- > >>> SHA512 (target/artifacts/hadoop-2.10.2-site.tar.gz) = > >> > >> > 3055a830003f5012660d92da68a317e15da5b73301c2c73cf618e724c67b7d830551b16928e0c28c10b66f04567e4b6f0b564647015bacc4677e232c0011537f > >> 1c1 > >> < SHA512 (hadoop-2.10.2-src.tar.gz) = > >> > >> > 483b6a4efd44234153e21ffb63a9f551530a1627f983a8837c655ce1b8ef13486d7178a7917ed3f35525c338e7df9b23404f4a1b0db186c49880448988b88600 > >> --- > >>> SHA512 (target/artifacts/hadoop-2.10.2-src.tar.gz) = > >> > >> > 483b6a4efd44234153e21ffb63a9f551530a1627f983a8837c655ce1b8ef13486d7178a7917ed3f35525c338e7df9b23404f4a1b0db186c49880448988b88600 > >> 1c1 > >> < SHA512 (hadoop-2.10.2.tar.gz) = > >> > >> > 13e95907073d815e3f86cdcc24193bb5eec0374239c79151923561e863326988c7f32a05fb7a1e5bc962728deb417f546364c2149541d6234221b00459154576 > >> --- > >>> SHA512 (target/artifacts/hadoop-2.10.2.tar.gz) = > >> > >> > 13e95907073d815e3f86cdcc24193bb5eec0374239c79151923561e863326988c7f32a05fb7a1e5bc962728deb417f546364c2149541d6234221b00459154576 > >> > >> However, checksums are correct. > >> > >> * Builds from source look good > >> - mvn clean install -DskipTests > >> - mvn clean package -Pdist -DskipTests -Dtar > -Dmaven.javadoc.skip=true > >> > >> * Rat check, if run before building from source locally, fails with > error: > >> > >> [ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:2.10.2 or one of > its > >> dependencies could not be resolved: Could not find artifact > >> org.apache.hadoop:hadoop-maven-plugins:jar:2.10.2 in central ( > >> https://repo.maven.apache.org/maven2) -> [Help 1] > >> [ERROR] > >> > >> However, once we build locally, rat check passes (because > >> hadoop-maven-plugins 2.1
Re: [VOTE] Release Apache Hadoop 3.3.3 (RC1)
+1 (binding) - Verified all checksums. - Verified all signatures. - Built from source, including native code on Linux. - Ran several examples successfully. Chris Nauroth On Mon, May 16, 2022 at 10:06 AM Chao Sun wrote: > +1 > > - Compiled from source > - Verified checksums & signatures > - Launched a pseudo HDFS cluster and ran some simple commands > - Ran full Spark tests with the RC > > Thanks Steve! > > Chao > > On Mon, May 16, 2022 at 2:19 AM Ayush Saxena wrote: > > > > +1, > > * Built from source. > > * Successful native build on Ubuntu 18.04 > > * Verified Checksums. > > > (CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz) > > * Verified Signature. > > * Successful RAT check > > * Ran basic HDFS shell commands. > > * Ran basic YARN shell commands. > > * Verified version in hadoop version command and UI > > * Ran some MR example Jobs. > > * Browsed UI(Namenode/Datanode/ResourceManager/NodeManager/HistoryServer) > > * Browsed the contents of Maven Artifacts. > > * Browsed the contents of the website. > > > > Thanx Steve for driving the release, Good Luck!!! > > > > -Ayush > > > > On Mon, 16 May 2022 at 08:20, Xiaoqiao He wrote: > > > > > +1(binding) > > > > > > * Verified signature and checksum of the source tarball. > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package > > > -DskipTests -Pnative -Pdist -Dtar`. > > > * Setup pseudo cluster with HDFS and YARN. > > > * Run simple FsShell - mkdir/put/get/mv/rm and check the result. > > > * Run example mr applications and check the result - Pi & wordcount. > > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager > etc. > > > > > > Thanks Steve for your work. > > > > > > - He Xiaoqiao > > > > > > On Mon, May 16, 2022 at 4:25 AM Viraj Jasani > wrote: > > > > > > > > +1 (non-binding) > > > > > > > > * Signature: ok > > > > * Checksum : ok > > > > * Rat check (1.8.0_301): ok > > > > - mvn clean apache-rat:check > > > > * Built from source (1.8.0_301): ok > > > > - mvn clean install -DskipTests > > > > * Built tar from source (1.8.0_301): ok > > > > - mvn clean package -Pdist -DskipTests -Dtar > -Dmaven.javadoc.skip=true > > > > > > > > HDFS, MapReduce and HBase (2.5) CRUD functional testing on > > > > pseudo-distributed mode looks good. > > > > > > > > > > > > On Wed, May 11, 2022 at 10:26 AM Steve Loughran > > > > > > > wrote: > > > > > > > > > I have put together a release candidate (RC1) for Hadoop 3.3.3 > > > > > > > > > > The RC is available at: > > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/ > > > > > > > > > > The git tag is release-3.3.3-RC1, commit d37586cbda3 > > > > > > > > > > The maven artifacts are staged at > > > > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1349/ > > > > > > > > > > You can find my public key at: > > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > > > Change log > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/CHANGELOG.md > > > > > > > > > > Release notes > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/RELEASENOTES.md > > > > > > > > > > There's a very small number of changes, primarily critical > > > code/packaging > > > > > issues and security fixes. > > > > > > > > > > * The critical fixes which shipped in the 3.2.3 release. > > > > > * CVEs in our code and dependencies > > > > > * Shaded client packaging issues. > > > > > * A switch from log4j to reload4j > > > > > > > > > > reload4j is an active fork of the log4j 1.17 library with the > classes > > > > > which contain CVEs removed. Even though hadoop never used those > > > classes, > > > > > they regularly raised alerts on security scans and concen from > users. > > > > > Switching to the forked project allows us to ship a secure logging > > > > > framework. It will complicate the builds of downstream > > > > > maven/ivy/gradle projects which exclude our log4j artifacts, as > they > > > > > need to cut the new dependency instead/as well. > > > > > > > > > > See the release notes for details. > > > > > > > > > > This is the second release attempt. It is the same git commit as > > > before, > > > > > but > > > > > fully recompiled with another republish to maven staging, which > has bee > > > > > verified by building spark, as well as a minimal test project. > > > > > > > > > > Please try the release and vote. The vote will run for 5 days. > > > > > > > > > > -Steve > > > > > > > > > > > - > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > - > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > >
Re: ElasticByteBufferPool is ever growing thus can cause memory leak.
Thanks for discussing this, Mukund. Another difference between the 2 classes is that ElasticByteBufferPool supports a caller's preference for either on-heap or off-heap buffers and internally calls either allocate or allocateDirect. Do you intend to preserve that functionality too in a single merged class? Chris Nauroth On Wed, Jan 12, 2022 at 1:25 AM Mukund Madhav Thakur wrote: > Hello Everyone, > I was just browsing through the code while doing my Vectored IO stuff. It > seems like ElasticByteBufferPool is an ever growing pool and memory is not > getting released as there is no WeakReference being maintained in the pool. > This can cause memory leaks in the production environment. This is widely > used in places like StripedReconstructor, > DFSStripedInputStream, BlockBlobAppendStream etc. > > I would suggest we use DirectBufferPool class for direct buffer pooling as > it already is keeping WeakReference for the buffers. Although, we will have > to make this implement the ByteBufferPool interface and implement the > corresponding methods. Happy to make the API changes once finalized. > > Thanks, > Mukund >
Re: [DISCUSS] Hadoop 3.3.2 release?
+1 Chao, thank you very much for volunteering on the release. Chris Nauroth On Tue, Sep 7, 2021 at 10:00 PM Igor Dvorzhak wrote: > +1 > > On Tue, Sep 7, 2021 at 10:06 AM Chao Sun wrote: > >> Hi all, >> >> It has been almost 3 months since the 3.3.1 release and branch-3.3 has >> accumulated quite a few commits (118 atm). In particular, Spark community >> recently found an issue which prevents one from using the shaded Hadoop >> client together with certain compression codecs such as lz4 and snappy >> codec. The details are recorded in HADOOP-17891 and SPARK-36669. >> >> Therefore, I'm wondering if anyone is also interested in a 3.3.2 release. >> If there is no objection, I'd like to volunteer myself for the work as >> well. >> >> Best Regards, >> Chao >> >
[jira] [Created] (HDFS-11995) HDFS Architecture documentation incorrectly describes writing to a local temporary file.
Chris Nauroth created HDFS-11995: Summary: HDFS Architecture documentation incorrectly describes writing to a local temporary file. Key: HDFS-11995 URL: https://issues.apache.org/jira/browse/HDFS-11995 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 3.0.0-alpha3 Reporter: Chris Nauroth The HDFS Architecture documentation has a section titled "Staging" that describes clients writing to a local temporary file first before interacting with the NameNode to allocate file metadata. This information is incorrect. (Perhaps it was correct a long time ago, but it is no longer accurate with respect to the current implementation.) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11063) Set NameNode RPC server handler thread name with more descriptive information about the RPC call.
Chris Nauroth created HDFS-11063:
Summary: Set NameNode RPC server handler thread name with more
descriptive information about the RPC call.
Key: HDFS-11063
URL: https://issues.apache.org/jira/browse/HDFS-11063
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Chris Nauroth
We often run {{jstack}} on a NameNode process as a troubleshooting step if it
is suffering high load or appears to be hanging. By reading the stack trace,
we can identify if a caller is blocked inside an expensive operation. This
would be even more helpful if we updated the RPC server handler thread name
with more descriptive information about the RPC call. This could include the
calling user, the called RPC method, and the most significant argument to that
method (most likely the path).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11034) Provide a command line tool to clear decommissioned DataNode information from the NameNode without restarting.
Chris Nauroth created HDFS-11034: Summary: Provide a command line tool to clear decommissioned DataNode information from the NameNode without restarting. Key: HDFS-11034 URL: https://issues.apache.org/jira/browse/HDFS-11034 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Chris Nauroth Information about decommissioned DataNodes remains tracked in the NameNode for the entire NameNode process lifetime. Currently, the only way to clear this information is to restart the NameNode. This issue proposes to add a way to clear this information online, without requiring a process restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-6277) WebHdfsFileSystem#toUrl does not perform character escaping for rename
[ https://issues.apache.org/jira/browse/HDFS-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-6277. - Resolution: Won't Fix This bug is present in the 1.x line, but not 2.x or 3.x. I'm resolving this as Won't Fix, because 1.x is no longer under active maintenance. > WebHdfsFileSystem#toUrl does not perform character escaping for rename > --- > > Key: HDFS-6277 > URL: https://issues.apache.org/jira/browse/HDFS-6277 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Ramya Sunil >Assignee: Chris Nauroth > > Found this issue while testing HDFS-6141. WebHdfsFileSystem#toUrl does not > perform character escaping for rename and causes the operation to fail. > This bug does not exist on 2.x > For e.g: > $ hadoop dfs -rmr 'webhdfs://:/tmp/test dirname with spaces' > Problem with Trash.Unexpected HTTP response: code=400 != 200, op=RENAME, > message=Bad Request. Consider using -skipTrash option > rmr: Failed to move to trash: webhdfs://:/tmp/test dirname > with spaces -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Permission bit 12 in getFileInfo response
Hello John,
That is the ACL bit. The NameNode toggles on the ACL bit in getFileInfo
responses for inodes that have ACL entries attached to them. On the client
side, this results in calls to FsPermission#getAclBit returning true.
The purpose of the ACL bit is to help client applications identify files and
directories that have ACL entries attached. One specific example where this is
useful is in the output of the file system shell "ls" command. (See
org.apache.hadoop.fs.shell.Ls#processPath.) If the ACL bit is turned on, then
this is how the shell decides to append a '+' character after the basic
permissions, so the end user knows that ACL entries are present. If the ACL
bit didn’t exist, then applications like this would have to be implemented with
a more costly FileSystem#getAclStatus call, in addition to the existing
getFileInfo RPC.
Test cases defined in FSAclBaseTest check for the presence of the ACL bit where
expected.
--Chris Nauroth
On 9/19/16, 10:55 AM, "John Zhuge" wrote:
Hi Gurus,
Does anyone know the meaning of bit 12 in the permission field of
"getFileInfo" response? To my understanding, the bit 9 is sticky bit, along
with the lower 9 bits for user/group/other.
In this following trace, the "perm" field is "4584", i.e., "10750" in oct:
16/09/15 15:54:56 TRACE ipc.ProtobufRpcEngine: 1: Response <-
NAMENODE:8020: getFileInfo {fs { fileType: IS_DIR path: "" length: 0
permission { perm: 4584 } owner: "USER" group: "supergroup"
modification_time: 1473884314570 access_time: 0 block_replication: 0
blocksize: 0 fileId: 8798130 childrenNum: 1 storagePolicy: 0 }}
Thanks,
John Zhuge
Software Engineer, Cloudera
Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Andrew, thanks for adding your perspective on this. What is a realistic strategy for us to evolve the HDFS audit log in a backward-compatible way? If the API is essentially any form of ad-hoc scripting, then for any proposed audit log format change, I can find a reason to veto it on grounds of backward incompatibility. - I can’t add a new field on the end, because that would break an awk script that uses $NF expecting to find a specific field. - I can’t prepend a new field, because that would break a "cut -f1" expecting to find the timestamp. - HDFS can’t add any new features, because someone might have written a script that does "exit 1" if it finds an unexpected RPC in the "cmd=" field. - Hadoop is not allowed to add full IPv6 support, because someone might have written a script that looks at the "ip=" field and parses it by IPv4 syntax. On the CLI, a potential solution for evolving the output is to preserve the old format by default and only enable the new format if the user explicitly passes a new argument. What should we do for the audit log? Configuration flags in hdfs-site.xml? (That of course adds its own brand of complexity.) I’m particularly interested to hear potential solutions from people like Andrew and Allen who have been most vocal about the need for a stable format. Without a solution, this unfortunately devolves into the format being frozen within a major release line. We could benefit from getting a patch on the compatibility doc that addresses the HDFS audit log specifically. --Chris Nauroth On 8/18/16, 8:47 AM, "Andrew Purtell" wrote: An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better. Where I work when we move from one Hadoop 2.x minor to another we always spend time updating our deployment plans, alerting, log scraping, and related things due to changes. Some are debatable as if qualifying for the 'incompatible' designation. I think the audit logging change that triggered this discussion is a good example of one that does. If you want to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control events.) One has to code regular expressions for parsing them and reverse engineer under what circumstances an audit line is emitted so you can make assumptions about what transpired. Change either and you might break someone's automation for meeting industry or legal compliance obligations. Not a trivial matter. If you don't operate Hadoop in production you might not realize the implications of such a change. Glad to see Hadoop has community diversity to recognize it in some cases. > On Aug 18, 2016, at 6:57 AM, Junping Du wrote: > > I think Allen's previous comments are very misleading. > In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.) should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5) marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of them are already released. > > Thanks, > > Junping > > From: Vinod Kumar Vavilapalli > Sent: Wednesday, August 17, 2016 9:29 PM > To: Allen Wittenauer > Cc: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org > Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1 > > I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously wasn’t there. > > Anyways, this shouldn’t be in any of branch-2.* as committers there clearly mentioned that this is an incompatible change. > > I am reverting the patch from branch-2* . > > Thanks > +Vinod > >> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer wrote: >> >> >> >> -1 >> >> HDFS-9395 is an incompatible change: >> >> a) Why is not marked as such in the changes file? >> b) Why is an incompatible change in a micro release, much less a minor? >> c) Where is the release note for this change? >> >> >>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli wrote: >>> >>> Hi all, >>> >>> I've created a release candidate RC1 for Apache Hadoop 2.7.3. >>> >>> As discussed before, this is the
Re: [DISCUSS] Retire BKJM from trunk?
I recommend including the BookKeeper community in this discussion. I’ve added their user@ and dev@ lists to this thread. I do not see BKJM being used in practice. Removing it from trunk would be attractive in terms of less code for Hadoop to maintain and build, but if we find existing users that want to keep it, I wouldn’t object. --Chris Nauroth On 7/26/16, 11:14 PM, "Vinayakumar B" wrote: Hi All, BKJM was Active and made much stable when the NameNode HA was implemented and there was no QJM implemented. Now QJM is present and is much stable which is adopted by many production environment. I wonder whether it would be a good time to retire BKJM from trunk? Are there any users of BKJM exists? -Vinay - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Apache MSDN Offer is Back
That definitely was possible under the old deal. You could go through the MSDN site and download an iso for various versions of Windows and run it under VirtualBox. The MSDN site also would furnish a license key that you could use to activate the machine. I haven't yet gone through this new process to see if anything has changed in the benefits. --Chris Nauroth From: Ravi Prakash mailto:ravihad...@gmail.com>> Date: Wednesday, July 20, 2016 at 12:04 PM To: Chris Nauroth mailto:cnaur...@hortonworks.com>> Cc: "common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>" mailto:common-...@hadoop.apache.org>>, "hdfs-dev@hadoop.apache.org<mailto:hdfs-dev@hadoop.apache.org>" mailto:hdfs-dev@hadoop.apache.org>>, "yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>" mailto:yarn-...@hadoop.apache.org>>, "mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>" mailto:mapreduce-...@hadoop.apache.org>> Subject: Re: Apache MSDN Offer is Back Thanks Chris! I did avail of the offer a few months ago, and wasn't able to figure out if a windows license was also available. I want to run windows inside a virtual machine on my Linux laptop, for the rare cases that there are patches that may affect that. Any clue if that is possible? Thanks Ravi On Tue, Jul 19, 2016 at 4:09 PM, Chris Nauroth mailto:cnaur...@hortonworks.com>> wrote: A few months ago, we learned that the offer for ASF committers to get an MSDN license had gone away. I'm happy to report that as of a few weeks ago, that offer is back in place. For more details, committers can check out https://svn.apache.org/repos/private/committers and read donated-licenses/msdn.txt. --Chris Nauroth
Apache MSDN Offer is Back
A few months ago, we learned that the offer for ASF committers to get an MSDN license had gone away. I'm happy to report that as of a few weeks ago, that offer is back in place. For more details, committers can check out https://svn.apache.org/repos/private/committers and read donated-licenses/msdn.txt. --Chris Nauroth
[jira] [Resolved] (HDFS-10546) hadoop-hdfs-native-client fails distro build when trying to copy libhdfs binaries.
[ https://issues.apache.org/jira/browse/HDFS-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-10546. -- Resolution: Duplicate I just realized this duplicates HDFS-10353, which fixed the problem in trunk. We just need to cherry-pick that patch down to branch-2 and branch-2.8. I'll cover it over there. > hadoop-hdfs-native-client fails distro build when trying to copy libhdfs > binaries. > -- > > Key: HDFS-10546 > URL: https://issues.apache.org/jira/browse/HDFS-10546 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Blocker > > During the distro build, hadoop-hdfs-native-client copies the built libhdfs > binary artifacts for inclusion in the distro. It references an incorrect > path though. The copy fails and the build aborts. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10546) hadoop-hdfs-native-client fails distro build when trying to copy libhdfs binaries.
Chris Nauroth created HDFS-10546: Summary: hadoop-hdfs-native-client fails distro build when trying to copy libhdfs binaries. Key: HDFS-10546 URL: https://issues.apache.org/jira/browse/HDFS-10546 Project: Hadoop HDFS Issue Type: Bug Components: build Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Blocker During the distro build, hadoop-hdfs-native-client copies the built libhdfs binary artifacts for inclusion in the distro. It references an incorrect path though. The copy fails and the build aborts. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
Interestingly, that FindBugs warning in hadoop-azure-datalake was not flagged during pre-commit before I committed HADOOP-12666. I'm going to propose that we address it in scope of HADOOP-12875. --Chris Nauroth On 6/10/16, 10:30 AM, "Apache Jenkins Server" wrote: >For more details, see >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/ > >No changes > > > > >-1 overall > > >The following subsystems voted -1: >findbugs unit > > >The following subsystems voted -1 but >were configured to be filtered/ignored: >cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace > > >The following subsystems are considered long running: >(runtime bigger than 1h 0m 0s) >unit > > >Specific tests: > >FindBugs : > > module:hadoop-tools/hadoop-azure-datalake > int value cast to float and then passed to Math.round in >org.apache.hadoop.hdfs.web.PrivateAzureDataLakeFileSystem$BatchByteArrayIn >putStream.getSplitSize(int) At PrivateAzureDataLakeFileSystem.java:and >then passed to Math.round in >org.apache.hadoop.hdfs.web.PrivateAzureDataLakeFileSystem$BatchByteArrayIn >putStream.getSplitSize(int) At PrivateAzureDataLakeFileSystem.java:[line >925] > >Failed junit tests : > > hadoop.hdfs.server.namenode.TestEditLog > hadoop.yarn.server.resourcemanager.TestClientRMTokens > hadoop.yarn.server.resourcemanager.TestAMAuthorization > hadoop.yarn.server.TestContainerManagerSecurity > hadoop.yarn.server.TestMiniYarnClusterNodeUtilization > hadoop.yarn.client.cli.TestLogsCLI > hadoop.yarn.client.api.impl.TestAMRMProxy > hadoop.yarn.client.api.impl.TestDistributedScheduling > hadoop.yarn.client.TestGetGroups > hadoop.mapreduce.tools.TestCLI > hadoop.mapred.TestMRCJCFileOutputCommitter > >Timed out junit tests : > > org.apache.hadoop.yarn.client.cli.TestYarnCLI > org.apache.hadoop.yarn.client.api.impl.TestAMRMClient > org.apache.hadoop.yarn.client.api.impl.TestYarnClient > org.apache.hadoop.yarn.client.api.impl.TestNMClient > > > cc: > > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/diff-compile-cc-root.txt [4.0K] > > javac: > > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/diff-compile-javac-root.txt [164K] > > checkstyle: > > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/diff-checkstyle-root.txt [16M] > > pylint: > > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/diff-patch-pylint.txt [16K] > > shellcheck: > > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/diff-patch-shellcheck.txt [20K] > > shelldocs: > > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/diff-patch-shelldocs.txt [16K] > > whitespace: > > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/whitespace-eol.txt [12M] > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/whitespace-tabs.txt [1.3M] > > findbugs: > > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/branch-findbugs-hadoop-tools_hadoop-azure-datalake-warnings.html >[8.0K] > > javadoc: > > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/diff-javadoc-javadoc-root.txt [2.3M] > > unit: > > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [144K] > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop- >yarn-server-resourcemanager.txt [60K] > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop- >yarn-server-tests.txt [268K] > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt >[908K] > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-ma >preduce-client-core.txt [56K] > >https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/58/artifact >/out/patch-unit-hadoop-mapreduce-proj
Re: INotify and atime
Hello Rahul,
I don't believe there is any fundamental difference in the handling of the
atime update for HDFS inotify. I'd expect you to receive an event with
EventType.METADATA and MetadataUpdateEvent.TIMES, and I wouldn't expect
that you'd need to save a new fsimage to trigger the notification.
There is a test in the Hadoop codebase that exercises inotify event
delivery on atime update: TestDFSInotifyEventInputStream#testBasic. You
might try looking at the source of that test and comparing it to what your
code is doing to see if there is any difference.
--Chris Nauroth
On 6/8/16, 2:17 PM, "rahul gidwani" wrote:
>I was testing Inotify locally and noticed that unless I save the FSImage I
>can't get a MetadataUpdateEvent to get fired.
>
>The test is pretty basic:
>
>I setup a minicluster with
>conf.setLong("dfs.access.time.precision", 1);
>
>create a file (verify I get a EventType.Create)
>
>I read that file and verify on the filesystem that the atime has changed
>for that file.
>I poll 1 minute for an event to get fired, nothing happens.
>I save the namespace for that fsImage, the event gets fired and I get a
>EventType.METADATA
>
>Is this logic correct? All the other events I have tested do not require
>saving the FSImage to get fired, is the atime event handled differently
>somehow?
>
>thank you
>rahul
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-10502) Enabled memory locking and now HDFS won't start up
[ https://issues.apache.org/jira/browse/HDFS-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-10502. -- Resolution: Invalid Hello [~machey]. I recommend taking these questions to the u...@hadoop.apache.org mailing list. We use JIRA for tracking confirmed bugs and feature requests. We use u...@hadoop.apache.org for usage advice and troubleshooting. Regarding whether or not this is a recommended approach, I think it depends on a few other factors. Is the intent to use these cached files from Hadoop workloads, such as MapReduce jobs or Hive queries? If not, then I wonder if your use case might be better served by something more directly focused on general caching use cases, such as Redis or memcached. If your use case does involve Hadoop integration, then certainly Centralized Cache Management is worth exploring. Regarding the timeouts, I can tell from the exception that this is the heartbeat RPC sent from the DataNode to the NameNode. I recommend investigating connectivity between the DataNode and the NameNode and examining the logs from both sides to try to determine if something is going wrong in the handling of the heartbeat message. On one hand, a heartbeat timeout is not an error condition that is specific to Centralized Cache Management. It could happen whether or not you're using that feature. On the other hand, the heartbeat message does contain some optional information about the state of cache capacity and current usage at the DataNode. That information would trigger special handling logic at the NameNode side, so I suppose there is a chance that something in that logic is hanging up the heartbeat handling. Investigating the logs might reveal more. u...@hadoop.apache.org would be a good forum for further discussion of both of these topics. > Enabled memory locking and now HDFS won't start up > -- > > Key: HDFS-10502 > URL: https://issues.apache.org/jira/browse/HDFS-10502 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Affects Versions: 2.7.2 > Environment: RHEL 6.8 >Reporter: Chris Machemer > > My goal is to speed up reads. I have about 500k small files (2k to 15k) and > I'm trying to use HDFS as a cache for serialized instances of java objects. > I've written the code to construct and serialize all the objects out to HDFS, > and am now hoping to improve read performance, because accessing the objects > from disk-based storage is proving to be too slow for my application's SLA's. > So my first question is, is using memory locking and hdfs cacheadmin pools > and directives the right way to go, to cache my objects into memory, or > should I create RAM disks, and do memory-based storage instead? > If hdfs cacheadmin is the way to go (it's the path I'm going down so far), > then I need to figure out if what's happening is a bug or if I've configured > something wrong, because when I start up HDFS with a gig of memory locked > (both in limits.d for ulimit -l and also in hdfs-site.xml) and the server > starts up, and presumably tries to cache things into memory, I get hours and > hours of timeouts in the logs like this: > 2016-06-08 07:42:50,856 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > IOException in offerService > java.net.SocketTimeoutException: Call From stgb-fe1.litle.com/10.1.9.66 to > localhost:8020 failed on socket timeout exception: > java.net.SocketTimeoutException: 6 millis timeout while waiting for > channel to be ready for read. ch : java.nio.channels.SocketChannel[connected > local=/127.0.0.1:51647 remote=localhost/127.0.0.1:8020]; For more details > see: http://wiki.apache.org/hadoop/SocketTimeout > at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:751) > at org.apache.hadoop.ipc.Client.call(Client.java:1479) > at org.apache.hadoop.ipc.Client.call(Client.java:1412) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) > at com.sun.proxy.$Proxy13.sendHeartbeat(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:153) > at > org.apache.h
Re: Another thought on client-side support of HDFS federation
Hello Tianyi HE, I noticed that a similar design for a federation proxying model has just been proposed on Apache JIRA HDFS-10467. You might want to join the conversation there. https://issues.apache.org/jira/browse/HDFS-10467 --Chris Nauroth On 5/2/16, 10:32 AM, "Colin McCabe" wrote: >Hi Tianyi HE, > >Thanks for sharing this! This reminds me of the httpfs daemon. This >daemon basically sits in front of an HDFS cluster and accepts requests, >which it serves by forwarding them to the underlying HDFS instance. >There is some documentation about it here: >https://hadoop.apache.org/docs/stable/hadoop-hdfs-httpfs/index.html > >Since httpfs uses an org.apache.hadoop.fs.FileSystem instance, it seems >like you could plug in the apache.hadoop.fs.viewfs.ViewFileSystem class >and be up and running with federation. I haven't tried this, but I >would expect that it would work, unless there are bugs in ViewFS itself. > >The big advantage of httpfs is that it provides a webhdfs-style REST >interface. As you said, this kind of interface makes it simple to use >any language with REST bindings, without worrying about using a thick >client. > >The big disadvantage of httpfs is that you must move both metadata and >data operations through the httpfs daemon. This could become a >performance bottleneck. It seems like you are concerned about this >bottleneck. > >We also have webhdfs. Unlike httpfs, webhdfs doesn't require all the >data to move through its daemon. With webhdfs, the client talks to >DataNodes directly. > >I wonder if extending httpfs or webhdfs would be a better approach than >starting from scratch. There is a maintenance burden for adding new >services and daemons. This was our motivation for removing hftp, for >example. It's certainly something to think about. > >best, >Colin > > >On Thu, Apr 28, 2016, at 17:55, 何天一 wrote: >> Hey guys, >> >> My associates have investigated HDFS federation recently, which, turns >> out >> to be a quite good solution for improving scalability on >> NameNode/DataNode >> side. >> >> However, we encountered some problem on client-side. Since: >> A) For historical reason, we use clients in multiple languages to access >> HDFS, (i.e. python-snakebite, or perhaps libhdfs++). So we either >> implement >> multiple versions of ViewFS or we give up the consistency view (which >>can >> be confusing to user). >> B) We have hadoop client configuration deployed on client nodes, which >>we >> do not have control over . Also, releasing new configuration could be a >> real heavy operation because it needs to be pushed to several thousand >>of >> nodes, as well as maintaining consistency (say a node is down throughout >> the operation, then come back online. it could still possess a stale >> version of configuration). >> >> So we intended to explore another solution to these problems, and came >>up >> with a proxy model. >> That is, build a RPC proxy in front of NameNodes. >> All clients talk to proxy when they need to consult NameNode, then proxy >> decide which NameNode should the request go to according to mount table. >> This solved our problem. All clients are seamlessly upgraded with >> federation support. >> We open sourced the proxy recently: https://github.com/bytedance/nnproxy >> (BTW, all kinds of feedbacks are welcomed) >> >> But there are still a few issues. For example, several modifications >> needs >> to be done inside hadoop ipc to support rpc forwarding. We released >>patch >> according to which with nnproxy project ( >> https://github.com/bytedance/nnproxy/tree/master/hadoop-patches). But it >> could be better to have these merged to apache trunk. Does someone think >> it's worth? >> >> >> -- >> Cheers, >> Tianyi HE >> (+86) 185 0042 4096 > >- >To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Cp command is not atomic
Hello Kun, You are correct that "hdfs dfs -cp" is not atomic, but the details of that are a bit different from what you described. For the example you gave, the sequence of events would be: 1. Open a.xml. 2. Create file b.xml._COPYING_. 3. Copy the bytes from a.xml to b.xml._COPYING_. 4. Rename b.xml._COPYING_ to b.xml. b.xml._COPYING_ is a temporary file. All the bytes are written to this location first. Only if the full copy is successful, it proceeds to step 4 to rename it to its final destination at b.xml. The rename is atomic, so overall, this has the effect that b.xml will never have partially-written data. Either the whole copy succeeds or the copy fails and b.xml doesn't exist. However, even though the rename is atomic, we can't claim the overall operation is atomic. For example, if the process dies between step 2 and step 3, then the command leaves a lingering side effect in the form of the b.xml._COPYING_ file. Perhaps it's sufficient for your use case that the final rename step is atomic. --Chris Nauroth On 5/25/16, 8:21 AM, "Kun Ren" wrote: >Hi Genius, > >If I understand correctly, the shell command "cp" for the HDFS is not >atomic, is that correct? > >For example: > >./bin/hdfs dfs -cp input/a.xml input/b.xml > >This command actually does 3 things, 1. read input/a.xml; 2. Create a new >file input/b.xml; 3. Write the content of a.xml to b.xml; > >When I looked at the code, and the client side actually does the 3 steps >and there are no lock between the 3 step, does it mean that the cp command >is not guaranteed atomic? > > >Thanks a lot for your reply. - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: HDFS Federation-- cross namenodes operations
Hello Kun, I replied to this same question on the user@ list. For usage questions like this, the user@ list is the correct forum. It's not necessary to cross-post to the dev@ lists. --Chris Nauroth On 5/25/16, 8:58 AM, "Kun Ren" wrote: >Hi Genius, > >Does HDFS Federation support the cross namenodes operations? > >For example: > >./bin/hdfs dfs -cp input1/a.xml input2/b.xml > >Supposed that input1 belongs namenode 1, and input 2 belongs namenode 2, >does Federation support this operation? And if not, why? > >Thanks. - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: ASF OS X Build Infrastructure
Hi Ravi, Something certainly seems off about that bootstrapping problem you encountered. :-) When I've done this, the artifact I downloaded was an .iso file, which I could then use to install a VirtualBox VM. I'm now tuned into the discussion Sean referenced about the ASF MSDN program. I'll send another update when I have something more specific to share. --Chris Nauroth From: Ravi Prakash mailto:ravihad...@gmail.com>> Date: Friday, May 20, 2016 at 4:56 PM To: Sean Busbey mailto:bus...@cloudera.com>> Cc: Chris Nauroth mailto:cnaur...@hortonworks.com>>, Steve Loughran mailto:ste...@hortonworks.com>>, Hadoop Common mailto:common-...@hadoop.apache.org>>, "mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>" mailto:mapreduce-...@hadoop.apache.org>>, "hdfs-dev@hadoop.apache.org<mailto:hdfs-dev@hadoop.apache.org>" mailto:hdfs-dev@hadoop.apache.org>>, "yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>" mailto:yarn-...@hadoop.apache.org>> Subject: Re: ASF OS X Build Infrastructure FWIW, I was able to get a response from the form last month. I was issued a new MSDN subscriber ID using which I could have downloaded Microsoft Visual Studio (and some other products, I think). I was interested in downloading an image of Windows to run in a VM, but the downloader is. wait for it. an exe file :-) Haven't gotten around to begging someone with a Windows OS to run that image downloader. On Fri, May 20, 2016 at 10:39 AM, Sean Busbey mailto:bus...@cloudera.com>> wrote: Some talk about the MSDN-for-committers program recently passed by on a private list. It's still active, it just changed homes within Microsoft. The info should still be in the committer repo. If something is amiss please let me know and I'll pipe up to the folks already plugged in to confirming it's active. On Fri, May 20, 2016 at 12:13 PM, Chris Nauroth mailto:cnaur...@hortonworks.com>> wrote: > It's very disappointing to see that vanish. I'm following up to see if I > can learn more about what happened or if I can do anything to help > reinstate it. > > --Chris Nauroth > > > > > On 5/20/16, 6:11 AM, "Steve Loughran" > mailto:ste...@hortonworks.com>> wrote: > >> >>> On 20 May 2016, at 10:40, Lars Francke >>> mailto:lars.fran...@gmail.com>> wrote: >>> >>>> >>>> Regarding lack of personal access to anything but Linux, I'll take >>>>this as >>>> an opportunity to remind everyone that ASF committers (not just >>>>limited to >>>> Hadoop committers) are entitled to a free MSDN license, which can get >>>>you >>>> a Windows VM for validating Windows issues and any patches that touch >>>> cross-platform concerns, like the native code. Contributors who are >>>>not >>>> committers still might struggle to get access to Windows, but all of us >>>> reviewing and committing patches do have access. >>>> >>> >>> Actually, from all I can tell this MSDN offer has been discontinued for >>> now. All the information has been removed from the committers repo. Do >>>you >>> have any more up to date information on this? >>> >> >> >>That's interesting. >> >>I did an SVN update and it went away..looks like something happened on >>April 26 >> >>No idea, though the svn log has a bit of detail >> >>- >>To unsubscribe, e-mail: >>mapreduce-dev-unsubscr...@hadoop.apache.org<mailto:mapreduce-dev-unsubscr...@hadoop.apache.org> >>For additional commands, e-mail: >>mapreduce-dev-h...@hadoop.apache.org<mailto:mapreduce-dev-h...@hadoop.apache.org> >> >> > > > - > To unsubscribe, e-mail: > common-dev-unsubscr...@hadoop.apache.org<mailto:common-dev-unsubscr...@hadoop.apache.org> > For additional commands, e-mail: > common-dev-h...@hadoop.apache.org<mailto:common-dev-h...@hadoop.apache.org> > -- busbey - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org<mailto:mapreduce-dev-unsubscr...@hadoop.apache.org> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org<mailto:mapreduce-dev-h...@hadoop.apache.org>
Re: ASF OS X Build Infrastructure
It's very disappointing to see that vanish. I'm following up to see if I can learn more about what happened or if I can do anything to help reinstate it. --Chris Nauroth On 5/20/16, 6:11 AM, "Steve Loughran" wrote: > >> On 20 May 2016, at 10:40, Lars Francke wrote: >> >>> >>> Regarding lack of personal access to anything but Linux, I'll take >>>this as >>> an opportunity to remind everyone that ASF committers (not just >>>limited to >>> Hadoop committers) are entitled to a free MSDN license, which can get >>>you >>> a Windows VM for validating Windows issues and any patches that touch >>> cross-platform concerns, like the native code. Contributors who are >>>not >>> committers still might struggle to get access to Windows, but all of us >>> reviewing and committing patches do have access. >>> >> >> Actually, from all I can tell this MSDN offer has been discontinued for >> now. All the information has been removed from the committers repo. Do >>you >> have any more up to date information on this? >> > > >That's interesting. > >I did an SVN update and it went away..looks like something happened on >April 26 > >No idea, though the svn log has a bit of detail > >- >To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org >For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10438) When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.
Chris Nauroth created HDFS-10438:
Summary: When NameNode HA is configured to use the lifeline RPC
server, it should log the address of that server.
Key: HDFS-10438
URL: https://issues.apache.org/jira/browse/HDFS-10438
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode
Reporter: KWON BYUNGCHANG
Assignee: Chris Nauroth
Priority: Minor
As reported by [~magnum]:
I have configured below
{code}
dfs.namenode.servicerpc-address.xdev.nn1=my.host.com:8040
dfs.namenode.lifeline.rpc-address.xdev.nn1=my.host.com:8041
{code}
servicerpc port is 8040, lifeline port is 8041.
however zkfc daemon is logging using servicerpc port.
It may cause confusion.
thank you.
{code}
2016-05-19 19:18:40,566 WARN ha.HealthMonitor
(HealthMonitor.java:doHealthChecks(207)) - Service health check failed for
NameNode at my.host.com/10.114.87.91:8040: The NameNode has no resources
available
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10437) ReconfigurationProtocol not covered by HDFSPolicyProvider.
Chris Nauroth created HDFS-10437:
Summary: ReconfigurationProtocol not covered by HDFSPolicyProvider.
Key: HDFS-10437
URL: https://issues.apache.org/jira/browse/HDFS-10437
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.8.0
Reporter: Chris Nauroth
The {{HDFSPolicyProvider}} class contains an entry for defining the security
policy of each HDFS RPC protocol interface. {{ReconfigurationProtocol}} is not
listed currently. This may indicate that reconfiguration functionality is not
working correctly in secured clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: ASF OS X Build Infrastructure
Allen, thank you for doing this. Regarding lack of personal access to anything but Linux, I'll take this as an opportunity to remind everyone that ASF committers (not just limited to Hadoop committers) are entitled to a free MSDN license, which can get you a Windows VM for validating Windows issues and any patches that touch cross-platform concerns, like the native code. Contributors who are not committers still might struggle to get access to Windows, but all of us reviewing and committing patches do have access. It has long been on my TODO list to set up similar Jenkins jobs for Windows, but it keeps slipping. I'll try once again to bump up priority. --Chris Nauroth On 5/19/16, 9:41 AM, "Allen Wittenauer" wrote: > > Some of you may not know that the ASF actually does have an OS X machine >(a Mac mini, so it¹s not a speed demon) in the build infrastructure. >While messing around with getting all? of the trunk jobs reconfigured to >do Java 8 and separate maven repos, I noticed that this box tends to sit >idle most of the day. Why take advantage of it? Therefore, I also setup >two jobs for us to use to help alleviate the ³I don¹t have access to >anything but Linux² excuse when writing code that may not work in a >portable manner. > >Jobs #1: > > https://builds.apache.org/view/H-L/view/Hadoop/job/Precommit-HADOOP-OSX > > This basically runs Apache Yetus precommit with quite a few of the >unnecessary tests disabled. For example, there¹s no point in running >checkstyle. Note that this job takes the *full* JIRA issue id as input. >So ŒHADOOP-9902¹ not Œ9902¹. This allows for one Jenkins job to be used >for all the Hadoop sub-projects (HADOOP, HDFS, MR, YARN). ³But my code >is on github and I don¹t want to upload a patch!² I haven¹t tested it, >but it should also take a URL, so just add a .diff to the end of your >github compare URL and put that in the issue box. It hypothetically >should work. > >Job #2: > > I¹m still hammering on this one because the email notifications aren¹t >working to my satisfaction plus we have some extremely Linux-specific >code in YARNŠ but > > > https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-trunk-osx-java8 >/ > > Š is a ³build the world² job similar to what is currently running under >the individual sub projects. (This actually makes it one of the few >³build everything² jobs we have running. Most of the other jobs only >build that particular sub project.). It does not run the full unit test >suite and it also does not build all of the native code. This gives us a >place to start on our journey of making Hadoop actually, truly run >everywhere. (Interesting side note: It¹s been *extremely* consistent in >what fails vs. the Linux build hosts.) > > At some point, likely after YETUS-390 is complete, I¹ll switch this job >over to be run by Apache Yetus in qbt mode so that it¹s actually easier >to track failures across all dirs. A huge advantage over raw maven >commands. > > Happy testing everyone. > > NOTE: if you don¹t have access to launch jobs on builds.apache.org, >you¹ll need to send a request to private@. The Apache Hadoop PMC has the >keys to give access to folks. > > > >- >To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Understood about the tests. --Chris Nauroth On 5/15/16, 7:30 AM, "Allen Wittenauer" wrote: > >> On May 14, 2016, at 3:11 PM, Chris Nauroth >>wrote: >> >> +1 (binding) >> >> -Tried a dry-run merge of HADOOP-12930 to trunk. >> -Successfully built distro on Windows. >> -Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs >> commands through Cygwin. >> -Reviewed documentation. >> >> Allen, thank you for the contribution. Would you please attach a full >> patch to HADOOP-12930 to check pre-commit results? > > > Nope. The whole reason this was done as a branch with multiple patches >was to prevent Jenkins from getting overwhelmed since it would trigger >full unit tests on pretty much the entire code base…. > >> While testing this, I discovered a bug in the distro build for Windows. >> Could someone please code review my patch on HADOOP-13149? > > Done! > >> >> --Chris Nauroth >> >> >> >> >> On 5/9/16, 1:26 PM, "Allen Wittenauer" wrote: >> >>> >>> Hey gang! >>> >>> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) >>>to >>> merge the HADOOP-12930 feature branch into trunk. This branch was >>> developed exclusively by me as per the discussion two months ago as a >>>way >>> to make what would be a rather large patch hopefully easier to review. >>> The vast majority of the branch is code movement in the same file, >>> additional license headers, maven assembly hooks for distribution, and >>> variable renames. Not a whole lot of new code, but a big diff file >>> none-the-less. >>> >>> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹ >>>commands >>> to allow for subcommands to be added or modified at runtime. This >>>allows >>> for individual users or entire sites to tweak the execution environment >>> to suit their local needs. For example, it has been a practice for >>>some >>> locations to change the distcp jar out for a custom one. Using this >>> functionality, it is possible that the Œhadoop distcp¹ command could >>>run >>> the local version without overwriting the bundled jar and for existing >>> documentation (read: results from Internet searches) to work as written >>> without modification. This has the potential to be a huge win, >>>especially >>> for: >>> >>> * advanced end users looking to supplement the Apache Hadoop >>>experience >>> * operations teams that may be able to leverage existing >>>documentation >>> without having to remain local ³exception² docs >>> * development groups wanting an easy way to trial experimental >>>features >>> >>> Additionally, this branch includes the following, related changes: >>> >>> * Adds the first unit tests for the Œhadoop¹ command >>> * Adds the infrastructure for hdfs script testing and the first >>> unit >>> test for the Œhdfs¹ command >>> * Modifies the hadoop-tools components to be dynamic rather >>> than hard >>> coded >>> * Renames the shell profiles for hdfs, mapred, and yarn to be >>> consistent with other bundled profiles, including the ones introduced >>>in >>> this branch >>> >>> Documentation, including a Œhello world¹-style example, is in the >>> UnixShellGuide markdown file. (Of course!) >>> >>> I am at ApacheCon this week if anyone wants to discuss in-depth. >>> >>> Thanks! >>> >>> P.S., >>> >>> There are still two open sub-tasks. These are blocked by other issues >>> so that we may add unit testing to the shell code in those respective >>> areas. I¹ll covert to full issues after HADOOP-12930 is closed. >>> >>> >>> - >>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >>> >>> >> > > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
+1 (binding) -Tried a dry-run merge of HADOOP-12930 to trunk. -Successfully built distro on Windows. -Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs commands through Cygwin. -Reviewed documentation. Allen, thank you for the contribution. Would you please attach a full patch to HADOOP-12930 to check pre-commit results? While testing this, I discovered a bug in the distro build for Windows. Could someone please code review my patch on HADOOP-13149? --Chris Nauroth On 5/9/16, 1:26 PM, "Allen Wittenauer" wrote: > > Hey gang! > > I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to >merge the HADOOP-12930 feature branch into trunk. This branch was >developed exclusively by me as per the discussion two months ago as a way >to make what would be a rather large patch hopefully easier to review. >The vast majority of the branch is code movement in the same file, >additional license headers, maven assembly hooks for distribution, and >variable renames. Not a whole lot of new code, but a big diff file >none-the-less. > > This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹ commands >to allow for subcommands to be added or modified at runtime. This allows >for individual users or entire sites to tweak the execution environment >to suit their local needs. For example, it has been a practice for some >locations to change the distcp jar out for a custom one. Using this >functionality, it is possible that the Œhadoop distcp¹ command could run >the local version without overwriting the bundled jar and for existing >documentation (read: results from Internet searches) to work as written >without modification. This has the potential to be a huge win, especially >for: > > * advanced end users looking to supplement the Apache Hadoop > experience > * operations teams that may be able to leverage existing > documentation >without having to remain local ³exception² docs > * development groups wanting an easy way to trial experimental > features > > Additionally, this branch includes the following, related changes: > > * Adds the first unit tests for the Œhadoop¹ command > * Adds the infrastructure for hdfs script testing and the first > unit >test for the Œhdfs¹ command > * Modifies the hadoop-tools components to be dynamic rather > than hard >coded > * Renames the shell profiles for hdfs, mapred, and yarn to be >consistent with other bundled profiles, including the ones introduced in >this branch > > Documentation, including a Œhello world¹-style example, is in the >UnixShellGuide markdown file. (Of course!) > >I am at ApacheCon this week if anyone wants to discuss in-depth. > > Thanks! > >P.S., > > There are still two open sub-tasks. These are blocked by other issues >so that we may add unit testing to the shell code in those respective >areas. I¹ll covert to full issues after HADOOP-12930 is closed. > > >- >To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-10373) HDFS ZKFC HealthMonitor Throw a Exception Cause AutoFailOver
[ https://issues.apache.org/jira/browse/HDFS-10373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-10373. -- Resolution: Invalid Hello [~piaoyu zhang]. This doesn't look like a bug. If ZKFC cannot contact its peer NameNode for a successful health check RPC, then an HA failover is the expected behavior. This looks like an operational problem in this environment that needs further investigation. "Connection reset by peer" means the remote end (the NameNode) closed out the socket before sending the expected response data. I recommend looking at the NameNode logs to see if anything unusual happened during the timeframe of the HA failover. If you need further assistance, then consider sending an email to u...@hadoop.apache.org. I hope this helps. > HDFS ZKFC HealthMonitor Throw a Exception Cause AutoFailOver > > > Key: HDFS-10373 > URL: https://issues.apache.org/jira/browse/HDFS-10373 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover >Affects Versions: 2.2.0 > Environment: CentOS6.5 Hadoop-2.2.0 >Reporter: zhangyubiao > Attachments: screenshot-1.png, 屏幕快照_2016-05-06_上午10.17.22.png > > > HDFS ZKFC HealthMonitor Throw a Exception > 2016-05-05 02:00:59,475 WARN org.apache.hadoop.ha.HealthMonitor: > Transport-level exception trying to monitor health of NameNode at > XXX-XXX-XXX-hadoop.jd.local/172.22.17 > 1.XX:8021: Failed on local exception: java.io.IOException: Connection reset > by peer; Host Details : local host is: > "XXX-XXX-XXX-hadoop.jd.local/172.22.171.XX"; destinat > ion host is: XXX-XXX-XXX-hadoop.jd.local":8021; > Cause HA AutoFailOver -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-10359) Allow trigger block report from all datanodes
[ https://issues.apache.org/jira/browse/HDFS-10359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-10359. -- Resolution: Won't Fix I think there is momentum in this conversation towards a "Won't Fix" resolution, so I'm resolving the issue now. [~Tao Jie], thank you for the discussion. Even though this didn't lead to an enhancement, we appreciate the participation. > Allow trigger block report from all datanodes > - > > Key: HDFS-10359 > URL: https://issues.apache.org/jira/browse/HDFS-10359 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.7.0, 2.6.1 >Reporter: Tao Jie > > Since we have HDFS-7278 allows trigger block report from one certain > datanode. It would be helpful to add a option to this command to trigger > block report from all datanodes. > Command maybe like this: > *hdfs dfsadmin -triggerBlockReport \[-incremental\] > * -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10356) Ozone: Container server needs enhancements to control of bind address for greater flexibility and testability.
Chris Nauroth created HDFS-10356:
Summary: Ozone: Container server needs enhancements to control of
bind address for greater flexibility and testability.
Key: HDFS-10356
URL: https://issues.apache.org/jira/browse/HDFS-10356
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Chris Nauroth
The container server, as implemented in class
{{org.apache.hadoop.ozone.container.common.transport.server.XceiverServer}},
currently does not offer the same degree of flexibility as our other RPC
servers for controlling the network interface and port used in the bind call.
There is no "bind-host" property, so it is not possible to control the exact
network interface selected. If the requested port is different from the actual
bound port (i.e. setting port to 0 in test cases), then there is no exposure of
that actual bound port to clients.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10351) Ozone: Optimize key writes to chunks by providing a bulk write implementation in ChunkOutputStream.
Chris Nauroth created HDFS-10351:
Summary: Ozone: Optimize key writes to chunks by providing a bulk
write implementation in ChunkOutputStream.
Key: HDFS-10351
URL: https://issues.apache.org/jira/browse/HDFS-10351
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Chris Nauroth
Assignee: Chris Nauroth
HDFS-10268 introduced the {{ChunkOutputStream}} class as part of end-to-end
integration of Ozone receiving key content and writing it to chunks in a
container. That patch provided an implementation of the mandatory single-byte
write method. We can improve performance by adding an implementation of the
bulk write method too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10349) StorageContainerManager fails to compile after merge of HDFS-10312 maxDataLength enforcement.
Chris Nauroth created HDFS-10349:
Summary: StorageContainerManager fails to compile after merge of
HDFS-10312 maxDataLength enforcement.
Key: HDFS-10349
URL: https://issues.apache.org/jira/browse/HDFS-10349
Project: Hadoop HDFS
Issue Type: Bug
Components: ozone
Reporter: Chris Nauroth
Assignee: Chris Nauroth
HDFS-10312 introduced enforcement of a configurable maximum data length while
deserializing large block reports. This change broke compilation of
{{StorageContainerManager}} on the HDFS-7240 feature branch, due to a
constructor signature change in {{DatanodeProtocolServerSideTranslatorPB}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: handlerCount
Hello, In general, configuration property default values will be defined in two places: 1) hdfs-default.xml, which defines the default property values when a deployment doesn't specifically set them and 2) DFSConfigKeys, a class that defines constant default values that the code uses if for some reason no default value is found during the configuration lookup. https://github.com/apache/hadoop/blob/rel/release-2.7.2/hadoop-hdfs-project /hadoop-hdfs/src/main/resources/hdfs-default.xml#L602-L606 https://github.com/apache/hadoop/blob/rel/release-2.7.2/hadoop-hdfs-project /hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java#L473-L 474 --Chris Nauroth On 4/28/16, 2:51 PM, "Kun Ren" wrote: >Hi Genius, > >I have a quick question: > >I remembered I saw the default value for HandlerCout is 10(The number of >Handler threads), but I can not find where it is defined in the source >code, could you please point to me where I can find it in the 2.7.2 >codebase? Thanks a lot.
[jira] [Resolved] (HDFS-10322) DomianSocket error lead to more and more DataNode thread waiting
[ https://issues.apache.org/jira/browse/HDFS-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-10322. -- Resolution: Duplicate [~chenfolin], thank you for investigating this further. I'm just updating status on this issue to indicate it's a duplicate of a prior issue. > DomianSocket error lead to more and more DataNode thread waiting > - > > Key: HDFS-10322 > URL: https://issues.apache.org/jira/browse/HDFS-10322 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: ChenFolin > Fix For: 2.6.4 > > > When open short read and a DomianSoket broken pipe error happened,The > Datanode will produce more and more waiting threads. > It is similar to Bug HADOOP-11802, but i do not think they are same problem, > because the DomainSocket thread is in Running state. > stack log: > "DataXceiver for client unix:/var/run/hadoop-hdfs/dn.50010 [Waiting for > operation #1]" daemon prio=10 tid=0x0278e000 nid=0x2bc6 waiting on > condition [0x7f2d6e4a5000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00061c493500> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:316) > at > org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.createNewMemorySegment(ShortCircuitRegistry.java:322) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitShm(DataXceiver.java:394) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitShm(Receiver.java:178) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:93) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > =DomianSocketWatcher > "Thread-759187" daemon prio=10 tid=0x0219c800 nid=0x8c56 runnable > [0x7f2dbe4cb000] >java.lang.Thread.State: RUNNABLE > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method) > at > org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52) > at > org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:474) > at java.lang.Thread.run(Thread.java:745) > ===datanode error log > ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: > datanode-:50010:DataXceiver error processing REQUEST_SHORT_CIRCUIT_SHM > operation src: unix:/var/run/hadoop-hdfs/dn.50010 dst: > java.net.SocketException: write(2) error: Broken pipe > at org.apache.hadoop.net.unix.DomainSocket.writeArray0(Native Method) > at org.apache.hadoop.net.unix.DomainSocket.access$300(DomainSocket.java:45) > at > org.apache.hadoop.net.unix.DomainSocket$DomainOutputStream.write(DomainSocket.java:601) > at > com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833) > at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843) > at > com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.sendShmSuccessResponse(DataXceiver.java:371) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitShm(DataXceiver.java:409) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitShm(Receiver.java:178) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:93) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HDFS-10322) DomianSocket error lead to more and more DataNode thread waiting
[ https://issues.apache.org/jira/browse/HDFS-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth reopened HDFS-10322: -- > DomianSocket error lead to more and more DataNode thread waiting > - > > Key: HDFS-10322 > URL: https://issues.apache.org/jira/browse/HDFS-10322 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: ChenFolin > Fix For: 2.6.4 > > > When open short read and a DomianSoket broken pipe error happened,The > Datanode will produce more and more waiting threads. > It is similar to Bug HADOOP-11802, but i do not think they are same problem, > because the DomainSocket thread is in Running state. > stack log: > "DataXceiver for client unix:/var/run/hadoop-hdfs/dn.50010 [Waiting for > operation #1]" daemon prio=10 tid=0x0278e000 nid=0x2bc6 waiting on > condition [0x7f2d6e4a5000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00061c493500> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:316) > at > org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.createNewMemorySegment(ShortCircuitRegistry.java:322) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitShm(DataXceiver.java:394) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitShm(Receiver.java:178) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:93) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > =DomianSocketWatcher > "Thread-759187" daemon prio=10 tid=0x0219c800 nid=0x8c56 runnable > [0x7f2dbe4cb000] >java.lang.Thread.State: RUNNABLE > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method) > at > org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52) > at > org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:474) > at java.lang.Thread.run(Thread.java:745) > ===datanode error log > ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: > datanode-:50010:DataXceiver error processing REQUEST_SHORT_CIRCUIT_SHM > operation src: unix:/var/run/hadoop-hdfs/dn.50010 dst: > java.net.SocketException: write(2) error: Broken pipe > at org.apache.hadoop.net.unix.DomainSocket.writeArray0(Native Method) > at org.apache.hadoop.net.unix.DomainSocket.access$300(DomainSocket.java:45) > at > org.apache.hadoop.net.unix.DomainSocket$DomainOutputStream.write(DomainSocket.java:601) > at > com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833) > at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843) > at > com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.sendShmSuccessResponse(DataXceiver.java:371) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitShm(DataXceiver.java:409) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitShm(Receiver.java:178) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:93) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-10312) Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction.
Chris Nauroth created HDFS-10312:
Summary: Large block reports may fail to decode at NameNode due to
64 MB protobuf maximum length restriction.
Key: HDFS-10312
URL: https://issues.apache.org/jira/browse/HDFS-10312
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Our RPC server caps the maximum size of incoming messages at 64 MB by default.
For exceptional circumstances, this can be uptuned using
{{ipc.maximum.data.length}}. However, for block reports, there is still an
internal maximum length restriction of 64 MB enforced by protobuf. (Sample
stack trace to follow in comments.) This issue proposes to apply the same
override to our block list decoding, so that large block reports can proceed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-10268) Ozone: end-to-end integration for create/get volumes, buckets and keys.
Chris Nauroth created HDFS-10268:
Summary: Ozone: end-to-end integration for create/get volumes,
buckets and keys.
Key: HDFS-10268
URL: https://issues.apache.org/jira/browse/HDFS-10268
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Chris Nauroth
Assignee: Chris Nauroth
The HDFS-7240 feature branch now has the building blocks required to enable
end-to-end functionality and testing for create/get volumes, buckets and keys.
The scope of this patch is to complete the necessary integration in
{{DistributedStorageHandler}} and related classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-10257) Quick Thread Local Storage set-up has a small flaw
[
https://issues.apache.org/jira/browse/HDFS-10257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-10257.
--
Resolution: Not A Problem
Great, thanks for confirming [~stevebovy]. I'll go ahead and close this issue.
> Quick Thread Local Storage set-up has a small flaw
> --
>
> Key: HDFS-10257
> URL: https://issues.apache.org/jira/browse/HDFS-10257
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 2.6.4
> Environment: Linux
>Reporter: Stephen Bovy
>Priority: Minor
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> In jni_helper.c in the getJNIEnvfunction
> The “THREAD_LOCAL_STORAGE_SET_QUICK(env);” Macro is in the wrong
> location;
> It should precede the “threadLocalStorageSet(env)” as follows ::
> THREAD_LOCAL_STORAGE_SET_QUICK(env);
> if (threadLocalStorageSet(env)) {
> return NULL;
> }
> AND IN “thread_local_storage.h” the macro:
> “THREAD_LOCAL_STORAGE_SET_QUICK”
> should be as follows ::
> #ifdef HAVE_BETTER_TLS
> #define THREAD_LOCAL_STORAGE_GET_QUICK() \
> static __thread JNIEnv *quickTlsEnv = NULL; \
> { \
> if (quickTlsEnv) { \
> return quickTlsEnv; \
> } \
> }
> #define THREAD_LOCAL_STORAGE_SET_QUICK(env) \
> { \
> quickTlsEnv = (env); \
> return env;
> }
> #else
> #define THREAD_LOCAL_STORAGE_GET_QUICK()
> #define THREAD_LOCAL_STORAGE_SET_QUICK(env)
> #endif
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-350) DFSClient more robust if the namenode is busy doing GC
[ https://issues.apache.org/jira/browse/HDFS-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-350. Resolution: Not A Problem I'm resolving this issue. In current versions, the client is more robust to this kind of failure. The RPC layer implements retry policies. Retried operations are handled gracefully using either an inherently idempotent implementation of the RPC or the retry cache for at-most-once execution. In the event of an extremely long GC, the client would either retry and succeed after completion of the GC, or in more extreme cases it would trigger an HA failover and the client would successfully issue its call to the the new active NameNode. > DFSClient more robust if the namenode is busy doing GC > -- > > Key: HDFS-350 > URL: https://issues.apache.org/jira/browse/HDFS-350 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: dhruba borthakur >Assignee: dhruba borthakur > > In the current code, if the client (writer) encounters an RPC error while > fetching a new block id from the namenode, it does not retry. It throws an > exception to the application. This becomes especially bad if the namenode is > in the middle of a GC and does not respond in time. The reason the client > throws an exception is because it does not know whether the namenode > successfully allocated a block for this file. > One possible enhancement would be to make the client retry the addBlock RPC > if needed. The client can send the block list that it currently has. The > namenode can match the block list send by the client with what it has in its > own metadata and then send back a new blockid (or a previously allocated > blockid that the client had not yet received because the earlier RPC > timedout). This will make the client more robust! > This works even when we support Appends because the namenode will *always* > verify that the client has the lease for the file in question. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9920) Stop tracking CHANGES.txt in the HDFS-7240 feature branch.
Chris Nauroth created HDFS-9920: --- Summary: Stop tracking CHANGES.txt in the HDFS-7240 feature branch. Key: HDFS-9920 URL: https://issues.apache.org/jira/browse/HDFS-9920 Project: Hadoop HDFS Issue Type: Sub-task Components: build Reporter: Chris Nauroth Assignee: Chris Nauroth Now that we have stopped tracking CHANGES.txt in our main branches, I'd like to do the same for the HDFS-7240 feature branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9907) Exclude Ozone protobuf-generated classes from Findbugs analysis.
Chris Nauroth created HDFS-9907: --- Summary: Exclude Ozone protobuf-generated classes from Findbugs analysis. Key: HDFS-9907 URL: https://issues.apache.org/jira/browse/HDFS-9907 Project: Hadoop HDFS Issue Type: Improvement Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Trivial Pre-commit runs on the HDFS-7240 feature branch are currently flagging Ozone protobuf-generated classes with warnings. These warnings aren't relevant, because we don't directly control the code generated by protoc. We can exclude these classes in the Findbugs configuration, just like we do for other existing protobuf-generated classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-9520) PeerCache evicts too frequently causing connection restablishments
[
https://issues.apache.org/jira/browse/HDFS-9520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-9520.
-
Resolution: Won't Fix
I'm resolving this as Won't Fix as per prior discussion. (Please feel free to
reopen if there are further thoughts on configuration tuning.)
> PeerCache evicts too frequently causing connection restablishments
> --
>
> Key: HDFS-9520
> URL: https://issues.apache.org/jira/browse/HDFS-9520
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: HDFS-9520.png
>
>
> Env: 20 node setup
> dfs.client.socketcache.capacity = 16
> Issue:
> ==
> Monitored PeerCache and it was evicting lots of connections during close. Set
> "dfs.client.socketcache.capacity=20" and tested again. Evictions still
> happened. Screenshot of profiler is attached in the JIRA.
> Workaround:
> ===
> Temp fix was to set "dfs.client.socketcache.capacity=1000" to prevent
> eviction.
> Added more debug logs revealed that multimap.size() was 40 instead of 20.
> LinkedListMultimap returns the total values instead of key size causing lots
> of evictions.
> {code}
>if (capacity == multimap.size()) {
> evictOldest();
> }
> {code}
> Should this be (capacity == multimap.keySet().size()) or is it expected that
> the "dfs.client.socketcache.capacity" be set to very high value?
> \cc [~gopalv], [~sseth]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8943) Read apis in ByteRangeInputStream does not read all the bytes specified when chunked transfer-encoding is used in the server
[
https://issues.apache.org/jira/browse/HDFS-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-8943.
-
Resolution: Won't Fix
[~cmccabe], thank you for the reminder. This is resolved as Won't Fix.
> Read apis in ByteRangeInputStream does not read all the bytes specified when
> chunked transfer-encoding is used in the server
>
>
> Key: HDFS-8943
> URL: https://issues.apache.org/jira/browse/HDFS-8943
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1
>Reporter: Shradha Revankar
>Assignee: Shradha Revankar
> Attachments: HDFS-8943.000.patch
>
>
> With the default Webhdfs server implementation the read apis in
> ByteRangeInputStream work as expected reading the correct number of bytes for
> these apis :
> {{public int read(byte b[], int off, int len)}}
> {{public int read(long position, byte[] buffer, int offset, int length)}}
> But when a custom Webhdfs server implementation is plugged in which uses
> chunked Transfer-encoding, these apis read only the first chunk. Simple fix
> would be to loop and read till bytes specified similar to {{readfully()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-9798) TestHdfsNativeCodeLoader fails
[
https://issues.apache.org/jira/browse/HDFS-9798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-9798.
-
Resolution: Duplicate
Hi [~ajisakaa]. A recent Yetus change is preventing pre-commit from building
libhadoop.so before running the HDFS tests. We're tracking the fix in
YETUS-281, and there is a patch in progress.
> TestHdfsNativeCodeLoader fails
> --
>
> Key: HDFS-9798
> URL: https://issues.apache.org/jira/browse/HDFS-9798
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira AJISAKA
>
> TestHdfsNativeCodeLoader fails intermittently in Jenkins.
> *
> https://builds.apache.org/job/PreCommit-HDFS-Build/14473/testReport/org.apache.hadoop.fs/TestHdfsNativeCodeLoader/testNativeCodeLoaded/
> *
> https://builds.apache.org/job/PreCommit-HDFS-Build/14475/testReport/org.apache.hadoop.fs/TestHdfsNativeCodeLoader/testNativeCodeLoaded/
> Error message
> {noformat}
> TestNativeCodeLoader: libhadoop.so testing was required, but libhadoop.so was
> not loaded. LD_LIBRARY_PATH =
> ${env.LD_LIBRARY_PATH}:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/native/target/usr/local/lib:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/../../hadoop-common-project/hadoop-common/target/native/target/usr/local/lib
> {noformat}
> Stacktrace
> {noformat}
> java.lang.AssertionError: TestNativeCodeLoader: libhadoop.so testing was
> required, but libhadoop.so was not loaded. LD_LIBRARY_PATH =
> ${env.LD_LIBRARY_PATH}:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/native/target/usr/local/lib:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/../../hadoop-common-project/hadoop-common/target/native/target/usr/local/lib
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.fs.TestHdfsNativeCodeLoader.testNativeCodeLoaded(TestHdfsNativeCodeLoader.java:46)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [Release thread] 2.8.0 release activities
FYI, I've just needed to raise HDFS-9761 to blocker status for the 2.8.0 release. --Chris Nauroth On 2/3/16, 6:19 PM, "Karthik Kambatla" wrote: >Thanks Vinod. Not labeling 2.8.0 stable sounds perfectly reasonable to me. >Let us not call it alpha or beta though, it is quite confusing. :) > >On Wed, Feb 3, 2016 at 8:17 PM, Gangumalla, Uma >wrote: > >> Thanks Vinod. +1 for 2.8 release start. >> >> Regards, >> Uma >> >> On 2/3/16, 3:53 PM, "Vinod Kumar Vavilapalli" >>wrote: >> >> >Seems like all the features listed in the Roadmap wiki are in. I¹m >>going >> >to try cutting an RC this weekend for a first/non-stable release off of >> >branch-2.8. >> > >> >Let me know if anyone has any objections/concerns. >> > >> >Thanks >> >+Vinod >> > >> >> On Nov 25, 2015, at 5:59 PM, Vinod Kumar Vavilapalli >> >> wrote: >> >> >> >> Branch-2.8 is created. >> >> >> >> As mentioned before, the goal on branch-2.8 is to put improvements / >> >>fixes to existing features with a goal of converging on an alpha >>release >> >>soon. >> >> >> >> Thanks >> >> +Vinod >> >> >> >> >> >>> On Nov 25, 2015, at 5:30 PM, Vinod Kumar Vavilapalli >> >>> wrote: >> >>> >> >>> Forking threads now in order to track all things related to the >> >>>release. >> >>> >> >>> Creating the branch now. >> >>> >> >>> Thanks >> >>> +Vinod >> >>> >> >>> >> >>>> On Nov 25, 2015, at 11:37 AM, Vinod Kumar Vavilapalli >> >>>> wrote: >> >>>> >> >>>> I think we¹ve converged at a high level w.r.t 2.8. And as I just >>sent >> >>>>out an email, I updated the Roadmap wiki reflecting the same: >> >>>>https://wiki.apache.org/hadoop/Roadmap >> >>>><https://wiki.apache.org/hadoop/Roadmap> >> >>>> >> >>>> I plan to create a 2.8 branch EOD today. >> >>>> >> >>>> The goal for all of us should be to restrict improvements & fixes >>to >> >>>>only (a) the feature-set documented under 2.8 in the RoadMap wiki >>and >> >>>>(b) other minor features that are already in 2.8. >> >>>> >> >>>> Thanks >> >>>> +Vinod >> >>>> >> >>>> >> >>>>> On Nov 11, 2015, at 12:13 PM, Vinod Kumar Vavilapalli >> >>>>>mailto:vino...@hortonworks.com>> wrote: >> >>>>> >> >>>>> - Cut a branch about two weeks from now >> >>>>> - Do an RC mid next month (leaving ~4weeks since branch-cut) >> >>>>> - As with 2.7.x series, the first release will still be called as >> >>>>>early / alpha release in the interest of >> >>>>> ‹ gaining downstream adoption >> >>>>> ‹ wider testing, >> >>>>> ‹ yet reserving our right to fix any inadvertent >>incompatibilities >> >>>>>introduced. >> >>>> >> >>> >> >> >> > >> >>
[jira] [Created] (HDFS-9711) Integrate CSRF prevention filter in WebHDFS.
Chris Nauroth created HDFS-9711: --- Summary: Integrate CSRF prevention filter in WebHDFS. Key: HDFS-9711 URL: https://issues.apache.org/jira/browse/HDFS-9711 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode, webhdfs Reporter: Chris Nauroth Assignee: Chris Nauroth HADOOP-12691 introduced a filter in Hadoop Common to help REST APIs guard against cross-site request forgery attacks. This issue tracks integration of that filter in WebHDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Hadoop encryption module as Apache Chimera incubator project
> My question is, could we consider to adopt the approach for libhadoop.so >library? This is something that I have proposed already in HADOOP-11127. There is not consensus on proceeding with it from the contributors in that discussion. There are some big challenges around how it would impact the release process. I also have not had availability to prototype an implementation to make a stronger case for feasibility. Kai, if this is something that you're interested in, then I encourage you to join the discussion in HADOOP-11127 or even pick up prototyping work if you'd like. Since we have that existing JIRA, let's keep this mail thread focused just on Chimera. Thank you! Uma and everyone, thank you for the proposal. +1 to proceed. --Chris Nauroth On 1/20/16, 11:16 PM, "Zheng, Kai" wrote: >Thanks Uma. > >I have a question by the way, it's not about Chimera project, but about >the mentioned advantage 1 and libhadoop.so installation problem. I copied >the saying as below for convenience. > >>>1. As Chimera embedded the native in jar (similar to Snappy java), it >>>solves the current issues in Hadoop that a HDFS client has to depend >>>libhadoop.so if the client needs to read encryption zone in HDFS. This >>>means a HDFS client may has to depend a Hadoop installation in local >>>machine. For example, HBase uses depends on HDFS client jar other than >>>a Hadoop installation and then has no access to libhadoop.so. So HBase >>>cannot use an encryption zone or it cause error. > >I believe Haifeng had mentioned the problem in a call when discussing >erasure coding work, but until now I got to understand what's the >problem and how Chimera or Snappy Java solved it. It looks like there can >be some thin clients that don't rely on Hadoop installation so no >libhadoop.so is available to use on the client host. The approach >mentioned here is to bundle the library file (*.so) into a jar and >dynamically extract the file when loading it. When no library file is >contained in the jar then it goes to the normal case, loading it from an >installation. It's smart and nice! My question is, could we consider to >adopt the approach for libhadoop.so library? It might be worth to discuss >because, we're bundling more and more things into the library (recently >we just put Intel ISA-L support into it), and such things may be desired >for such clients. It may also be helpful for development, because >sometimes when run unit tests that involve native codes, some error may >happen and complain no place to find libhadoop.so. Thanks. > >Regards, >Kai > >-Original Message- >From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] >Sent: Thursday, January 21, 2016 11:20 AM >To: hdfs-dev@hadoop.apache.org >Subject: Re: Hadoop encryption module as Apache Chimera incubator project > >Hi All, >Thanks Andrew, ATM, Yi, Kai, Larry. Thanks Haifeng on clarifying release >stuff. > >Please find my responses below. > >Andrew wrote: >If it becomes part of Apache Commons, could we make Chimera a separate >JAR? We have real difficulties bumping dependency versions right now, so >ideally we don't need to bump our existing Commons dependencies to use >Chimera. >[UMA] Yes, We plan to make separate Jar. > >Andrew wrote: >With this refactoring, do we have confidence that we can get our desired >changes merged and released in a timely fashion? e.g. if we find another >bug like HADOOP-11343, we'll first need to get the fix into Chimera, have >a new Chimera release, then bump Hadoop's Chimera dependency. This also >relates to the previous point, it's easier to do this dependency bump if >Chimera is a separate JAR. >[UMA] Yes and the main target users for this project is Hadoop and Spark >right now. >So, Hadoop requirements would be the priority tasks for it. > > >ATM wrote: >Uma, would you be up for approaching the Apache Commons folks saying that >you'd like to contribute Chimera? I'd recommend saying that Hadoop and >Spark are both on board to depend on this. >[UMA] Yes, will do that. > > >Kai wrote: >Just a question. Becoming a separate jar/module in Apache Commons means >Chimera or the module can be released separately or in a timely manner, >not coupling with other modules for release in the project? Thanks. > >[Haifeng] From apache commons project web (https://commons.apache.org/), >we see there is already a long list of components in its Apache Commons >Proper list. Each component has its own release version and date. To join >and be one of the list is the target. > >Larry wrote: >If what we are looking for is some level of autonomy then it would need &
[jira] [Reopened] (HDFS-6255) fuse_dfs will not adhere to ACL permissions in some cases
[
https://issues.apache.org/jira/browse/HDFS-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth reopened HDFS-6255:
-
I have a theory about what is happening. fuse_dfs is not specifically made
aware of the HDFS ACLs. It only has visibility into the basic permissions. In
the case of an ACL entry that widens access (i.e. grant access to a specific
named user or group), then if FUSE itself is enforcing access based solely on
permissions, it might block access at the FUSE layer before even delegating to
the NameNode. This would be a limitation in granting access via ACLs through
fuse_dfs, but it would not be a security hole. (The problem can only make
access more restrictive, not more relaxed.)
I tried to confirm this in the FUSE code, but I wasn't successful, and I don't
have time to look deeper right now. I'm seeing some comments from various
sources that FUSE is unaware of POSIX ACLs, but can be made aware of xattrs.
This might mean there is a possibility of making it work with some code changes
in fuse_dfs.
I'm not entirely sure this is feasible yet, but I'm going to reopen the issue
and mark it as a new feature request.
> fuse_dfs will not adhere to ACL permissions in some cases
> -
>
> Key: HDFS-6255
> URL: https://issues.apache.org/jira/browse/HDFS-6255
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fuse-dfs
>Affects Versions: 3.0.0, 2.4.0
> Reporter: Stephen Chu
>Assignee: Chris Nauroth
>
> As hdfs user, I created a directory /tmp/acl_dir/ and set permissions to 700.
> Then I set a new acl group:jenkins:rwx on /tmp/acl_dir.
> {code}
> jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -getfacl /tmp/acl_dir
> # file: /tmp/acl_dir
> # owner: hdfs
> # group: supergroup
> user::rwx
> group::---
> group:jenkins:rwx
> mask::rwx
> other::---
> {code}
> Through the FsShell, the jenkins user can list /tmp/acl_dir as well as create
> a file and directory inside.
> {code}
> [jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -touchz /tmp/acl_dir/testfile1
> [jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -mkdir /tmp/acl_dir/testdir1
> hdfs dfs -ls /tmp/acl[jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -ls /tmp/acl_dir/
> Found 2 items
> drwxr-xr-x - jenkins supergroup 0 2014-04-17 19:11
> /tmp/acl_dir/testdir1
> -rw-r--r-- 1 jenkins supergroup 0 2014-04-17 19:11
> /tmp/acl_dir/testfile1
> [jenkins@hdfs-vanilla-1 ~]$
> {code}
> However, as the same jenkins user, when I try to cd into /tmp/acl_dir using a
> fuse_dfs mount, I get permission denied. Same permission denied when I try to
> create or list files.
> {code}
> [jenkins@hdfs-vanilla-1 tmp]$ ls -l
> total 16
> drwxrwx--- 4 hdfsnobody 4096 Apr 17 19:11 acl_dir
> drwx-- 2 hdfsnobody 4096 Apr 17 18:30 acl_dir_2
> drwxr-xr-x 3 mapred nobody 4096 Mar 11 03:53 mapred
> drwxr-xr-x 4 jenkins nobody 4096 Apr 17 07:25 testcli
> -rwx-- 1 hdfsnobody0 Apr 7 17:18 tf1
> [jenkins@hdfs-vanilla-1 tmp]$ cd acl_dir
> bash: cd: acl_dir: Permission denied
> [jenkins@hdfs-vanilla-1 tmp]$ touch acl_dir/testfile2
> touch: cannot touch `acl_dir/testfile2': Permission denied
> [jenkins@hdfs-vanilla-1 tmp]$ mkdir acl_dir/testdir2
> mkdir: cannot create directory `acl_dir/testdir2': Permission denied
> [jenkins@hdfs-vanilla-1 tmp]$
> {code}
> The fuse_dfs debug output doesn't show any error for the above operations:
> {code}
> unique: 18, opcode: OPENDIR (27), nodeid: 2, insize: 48
>unique: 18, success, outsize: 32
> unique: 19, opcode: READDIR (28), nodeid: 2, insize: 80
> readdir[0] from 0
>unique: 19, success, outsize: 312
> unique: 20, opcode: GETATTR (3), nodeid: 2, insize: 56
> getattr /tmp
>unique: 20, success, outsize: 120
> unique: 21, opcode: READDIR (28), nodeid: 2, insize: 80
>unique: 21, success, outsize: 16
> unique: 22, opcode: RELEASEDIR (29), nodeid: 2, insize: 64
>unique: 22, success, outsize: 16
> unique: 23, opcode: GETATTR (3), nodeid: 2, insize: 56
> getattr /tmp
>unique: 23, success, outsize: 120
> unique: 24, opcode: GETATTR (3), nodeid: 3, insize: 56
> getattr /tmp/acl_dir
>unique: 24, success, outsize: 120
> unique: 25, opcode: GETATTR (3), nodeid: 3, insize: 56
> getattr /tmp/acl_dir
>unique: 25, success, outsize: 120
> unique: 26, opcode: GETATTR (3), nodeid: 3, insize: 56
> getattr /tmp/acl_dir
>unique: 26, success, outsize: 120
> unique: 27, opcode: GETATTR (3), nodeid: 3, insize: 56
> getattr /tmp/acl_dir
>unique: 27, success, outsize: 120
> unique: 28, opcode: GETATT
[jira] [Reopened] (HDFS-9569) Log the name of the fsimage being loaded for better supportability
[
https://issues.apache.org/jira/browse/HDFS-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth reopened HDFS-9569:
-
I have reverted this patch from trunk, branch-2, branch-2.8 and branch-2.7.
This patch introduced a test failure in
{{TestDFSUpgradeFromImage#testUpgradeFromRel2ReservedImage}}. The test expects
to see an {{IllegalArgumentException}}, and then retry the upgrade with the
option to rename reserved paths. After this patch, the error handling masked
the {{IllegalArgumentException}}, so the test no longer worked as expected.
> Log the name of the fsimage being loaded for better supportability
> --
>
> Key: HDFS-9569
> URL: https://issues.apache.org/jira/browse/HDFS-9569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.3
>
> Attachments: HDFS-9569.001.patch
>
>
> When NN starts to load fsimage, it does
> {code}
> void loadFSImageFile(FSNamesystem target, MetaRecoveryContext recovery,
> FSImageFile imageFile, StartupOption startupOption) throws IOException {
> LOG.debug("Planning to load image :\n" + imageFile);
> ..
> long txId = loader.getLoadedImageTxId();
> LOG.info("Loaded image for txid " + txId + " from " + curFile);
> {code}
> A debug msg is issued at the beginning with the fsimage file name, then at
> the end an info msg is issued after loading.
> If the fsimage loading failed due to corrupted fsimage (see HDFS-9406), we
> don't see the first msg. It'd be helpful to always be able to see from NN
> logs what fsimage file it's loading.
> Two improvements:
> 1. Change the above debug to info
> 2. If exception happens when loading fsimage, be sure to report the fsimage
> name being loaded in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9572) Prevent DataNode log spam if a client connects on the data transfer port but sends no data.
Chris Nauroth created HDFS-9572: --- Summary: Prevent DataNode log spam if a client connects on the data transfer port but sends no data. Key: HDFS-9572 URL: https://issues.apache.org/jira/browse/HDFS-9572 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: Chris Nauroth Assignee: Chris Nauroth Monitoring tools may choose to check liveness of the DataNode's data transfer port by connecting to it. The monitoring tool will close the connection immediately after establishment without sending any data. When this happens, the DataNode encounters an unexpected EOF and logs a full stack trace. This creates unneeded noise in the logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9552) Document types of permission checks performed for HDFS operations.
Chris Nauroth created HDFS-9552: --- Summary: Document types of permission checks performed for HDFS operations. Key: HDFS-9552 URL: https://issues.apache.org/jira/browse/HDFS-9552 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Reporter: Chris Nauroth Assignee: Chris Nauroth The HDFS permissions guide discusses our use of a POSIX-like model with read, write and execute permissions associated with users, groups and the catch-all other class. However, there is no documentation that describes exactly what permission checks are performed by user-facing HDFS operations. This is a frequent source of questions, so it would be good to document this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9534) Add CLI command to clear storage policy from a path.
Chris Nauroth created HDFS-9534:
---
Summary: Add CLI command to clear storage policy from a path.
Key: HDFS-9534
URL: https://issues.apache.org/jira/browse/HDFS-9534
Project: Hadoop HDFS
Issue Type: Improvement
Components: tools
Reporter: Chris Nauroth
The {{hdfs storagepolicies}} command has sub-commands for {{-setStoragePolicy}}
and {{-getStoragePolicy}} on a path. However, there is no
{{-removeStoragePolicy}} to remove a previously set storage policy on a path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Outdated "HDFS Architecture" page
Hello Dmitry, Thank you for reporting this. You're absolutely right. That page is out-of-date with respect to the current architecture. I filed JIRA HDFS-9505 to track improving this documentation. If you have any desire to contribute a documentation patch for this, then please feel free to assign the JIRA to yourself. --Chris Nauroth On 12/3/15, 12:15 PM, "Dmitry Simonov" wrote: >Hello! > >Excuse me if I'm writing to the wrong mailing list. > >The information at >http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Hdfs >Design.html >seems outdated: > >1. "There is a plan to support appending-writes to files in the future." > >Append was implemented in HADOOP-1700, wasn't it? > >2. "HDFS does not yet implement user quotas or access permissions" > >It contradicts to >https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Hdf >sQuotaAdminGuide.html >and >https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Hdf >sPermissionsGuide.html > >3. "Currently, automatic restart and failover of the NameNode software to >another machine is not supported." - but there is High Availability >feature. > >-- >Best Regards, >Dmitry Simonov
[jira] [Created] (HDFS-9505) HDFS Architecture documentation needs to be refreshed.
Chris Nauroth created HDFS-9505: --- Summary: HDFS Architecture documentation needs to be refreshed. Key: HDFS-9505 URL: https://issues.apache.org/jira/browse/HDFS-9505 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Reporter: Chris Nauroth Priority: Minor The HDFS Architecture document is out of date with respect to the current design of the system. http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html There are multiple false statements and omissions of recent features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-9495) Data node opens random port for HTTPServer, not configurable
[ https://issues.apache.org/jira/browse/HDFS-9495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-9495. - Resolution: Duplicate Hello, [~neha.bathra]. This issue is tracked in HDFS-9049, so I'm resolving this one as a duplicate. > Data node opens random port for HTTPServer, not configurable > > > Key: HDFS-9495 > URL: https://issues.apache.org/jira/browse/HDFS-9495 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: neha > > Data node opens random port for HTTP Server which is not configurable > currently. Better to make it configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-9471) Webhdfs not working with shell command when kerberos security+https is enabled.
[
https://issues.apache.org/jira/browse/HDFS-9471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-9471.
-
Resolution: Not A Problem
[~surendrasingh], that's a good point about the documentation. I filed
HDFS-9483 to track a documentation improvement. If you're interested in
providing the documentation, please feel free to pick up that one. I'm going
to resolve this one.
> Webhdfs not working with shell command when kerberos security+https is
> enabled.
> ---
>
> Key: HDFS-9471
> URL: https://issues.apache.org/jira/browse/HDFS-9471
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Blocker
> Attachments: HDFS-9471.01.patch
>
>
> *Client exception*
> {code}
> secure@host85:/opt/hdfsdata/HA/install/hadoop/namenode/bin> ./hdfs dfs -ls
> webhdfs://x.x.x.x:50070/test
> 15/11/25 18:46:55 ERROR web.WebHdfsFileSystem: Unable to get HomeDirectory
> from original File System
> java.net.SocketException: Unexpected end of file from server
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:792)
> {code}
> *Exception in namenode log*
> {code}
> 2015-11-26 11:03:18,231 WARN org.mortbay.log: EXCEPTION
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
> at
> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
> at sun.security.ssl.InputRecord.read(InputRecord.java:527)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:961)
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1391)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1375)
> at
> org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {code}
> This is because URL schema hard coded in
> {{WebHdfsFileSystem.getTransportScheme()}}.
> {code}
> /**
>* return the underlying transport protocol (http / https).
>*/
> protected String getTransportScheme() {
> return "http";
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9483) Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS.
Chris Nauroth created HDFS-9483: --- Summary: Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Key: HDFS-9483 URL: https://issues.apache.org/jira/browse/HDFS-9483 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Reporter: Chris Nauroth If WebHDFS is secured with SSL, then you can use "swebhdfs" as the scheme in a URL to access it. The current documentation does not state this anywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Looking to a 2.8.0 release
+1. Thanks, Vinod. --Chris Nauroth On 11/25/15, 1:45 PM, "Vinod Kumar Vavilapalli" wrote: >Okay, tx for this clarification Chris! I dug more into this and now >realized the actual scope of this. Given the the limited nature of this >feature (non-Namenode etc) and the WIP nature of the larger umbrella >HADOOP-11744, we will ship the feature but I’ll stop calling this out as >a notable feature. > >Thanks >+Vinod > > >> On Nov 25, 2015, at 12:04 PM, Chris Nauroth >>wrote: >> >> Hi Vinod, >> >> The HDFS-8155 work is complete in branch-2 already, so feel free to >> include it in the roadmap. >> >> For those watching the thread that aren't familiar with HDFS-8155, I >>want >> to call out that it was a client-side change only. The WebHDFS client >>is >> capable of obtaining OAuth2 tokens and passing them along in its HTTP >> requests. The NameNode and DataNode server side currently do not have >>any >> support for OAuth2, so overall, this feature is only useful in some very >> unique deployment architectures right now. This is all discussed >> explicitly in documentation committed with HDFS-8155, but I wanted to >> prevent any mistaken assumptions for people only reading this thread. >> >> --Chris Nauroth >> >> >> >> >> On 11/25/15, 11:08 AM, "Vinod Kumar Vavilapalli" >> wrote: >> >>> This is the current state from the feedback I gathered. >>> - Support priorities across applications within the same queue >>>YARN-1963 >>> ‹ Can push as an alpha / beta feature per Sunil >>> - YARN-1197 Support changing resources of an allocated container: >>> ‹ Can push as an alpha/beta feature per Wangda >>> - YARN-3611 Support Docker Containers In LinuxContainerExecutor: Well >>> most of it anyways. >>> ‹ Can push as an alpha feature. >>> - YARN Timeline Service v1.5 - YARN-4233 >>> ‹ Should include per Li Lu >>> - YARN Timeline Service Next generation: YARN-2928 >>> ‹ Per analysis from Sangjin, drop this from 2.8. >>> >>> One open feature status >>> - HDFS-8155Support OAuth2 in WebHDFS: Alpha / Early feature? >>> >>> Updated the Roadmap wiki with the same. >>> >>> Thanks >>> +Vinod >>> >>>> On Nov 13, 2015, at 12:12 PM, Sangjin Lee wrote: >>>> >>>> I reviewed the current state of the YARN-2928 changes regarding its >>>> impact >>>> if the timeline service v.2 is disabled. It does appear that there >>>>are a >>>> lot of things that still do get created and enabled unconditionally >>>> regardless of configuration. While this is understandable when we were >>>> working to implement the feature, this clearly needs to be cleaned up >>>>so >>>> that when disabled the timeline service v.2 doesn't impact other >>>>things. >>>> >>>> I filed a JIRA for that work: >>>> https://issues.apache.org/jira/browse/YARN-4356 >>>> >>>> We need to complete it before we can merge. >>>> >>>> Somewhat related is the status of the configuration and what it means >>>>in >>>> various contexts (client/app-side vs. server-side, v.1 vs. v.2, >>>>etc.). I >>>> know there is an ongoing discussion regarding YARN-4183. We'll need to >>>> reflect the outcome of that discussion. >>>> >>>> My overall impression of whether this can be done for 2.8 is that it >>>> looks >>>> rather challenging given the suggested timeframe. We also need to >>>> complete >>>> several major tasks before it is ready. >>>> >>>> Sangjin >>>> >>>> >>>> On Wed, Nov 11, 2015 at 5:49 PM, Sangjin Lee wrote: >>>> >>>>> >>>>> On Wed, Nov 11, 2015 at 12:13 PM, Vinod Vavilapalli < >>>>> vino...@hortonworks.com> wrote: >>>>> >>>>>> ‹ YARN Timeline Service Next generation: YARN-2928: Lots of >>>>>> momentum, >>>>>> but clearly a work in progress. Two options here >>>>>> ‹ If it is safe to ship it into 2.8 in a disable manner, we >>>>>>can >>>>>> get the early code into trunk and all the way int o2.8. >>>>>> ‹ If it is not safe, it organically rolls over into 2.9 >>>>>> >>>>> >>>>> I'll review the changes on YARN-2928 to see what impact it has (if >>>>> any) if >>>>> the timeline service v.2 is disabled. >>>>> >>>>> Another condition for it to make 2.8 is whether the branch will be >>>>>in a >>>>> shape in a couple of weeks such that it adds value for folks that >>>>>want >>>>> to >>>>> test it. Hopefully it will become clearer soon. >>>>> >>>>> Sangjin >>>>> >>> >> >> > >
Re: [DISCUSS] Looking to a 2.8.0 release
Hi Vinod, The HDFS-8155 work is complete in branch-2 already, so feel free to include it in the roadmap. For those watching the thread that aren't familiar with HDFS-8155, I want to call out that it was a client-side change only. The WebHDFS client is capable of obtaining OAuth2 tokens and passing them along in its HTTP requests. The NameNode and DataNode server side currently do not have any support for OAuth2, so overall, this feature is only useful in some very unique deployment architectures right now. This is all discussed explicitly in documentation committed with HDFS-8155, but I wanted to prevent any mistaken assumptions for people only reading this thread. --Chris Nauroth On 11/25/15, 11:08 AM, "Vinod Kumar Vavilapalli" wrote: >This is the current state from the feedback I gathered. > - Support priorities across applications within the same queue YARN-1963 >‹ Can push as an alpha / beta feature per Sunil > - YARN-1197 Support changing resources of an allocated container: >‹ Can push as an alpha/beta feature per Wangda > - YARN-3611 Support Docker Containers In LinuxContainerExecutor: Well >most of it anyways. >‹ Can push as an alpha feature. > - YARN Timeline Service v1.5 - YARN-4233 >‹ Should include per Li Lu > - YARN Timeline Service Next generation: YARN-2928 >‹ Per analysis from Sangjin, drop this from 2.8. > >One open feature status > - HDFS-8155Support OAuth2 in WebHDFS: Alpha / Early feature? > >Updated the Roadmap wiki with the same. > >Thanks >+Vinod > >> On Nov 13, 2015, at 12:12 PM, Sangjin Lee wrote: >> >> I reviewed the current state of the YARN-2928 changes regarding its >>impact >> if the timeline service v.2 is disabled. It does appear that there are a >> lot of things that still do get created and enabled unconditionally >> regardless of configuration. While this is understandable when we were >> working to implement the feature, this clearly needs to be cleaned up so >> that when disabled the timeline service v.2 doesn't impact other things. >> >> I filed a JIRA for that work: >> https://issues.apache.org/jira/browse/YARN-4356 >> >> We need to complete it before we can merge. >> >> Somewhat related is the status of the configuration and what it means in >> various contexts (client/app-side vs. server-side, v.1 vs. v.2, etc.). I >> know there is an ongoing discussion regarding YARN-4183. We'll need to >> reflect the outcome of that discussion. >> >> My overall impression of whether this can be done for 2.8 is that it >>looks >> rather challenging given the suggested timeframe. We also need to >>complete >> several major tasks before it is ready. >> >> Sangjin >> >> >> On Wed, Nov 11, 2015 at 5:49 PM, Sangjin Lee wrote: >> >>> >>> On Wed, Nov 11, 2015 at 12:13 PM, Vinod Vavilapalli < >>> vino...@hortonworks.com> wrote: >>> >>>>‹ YARN Timeline Service Next generation: YARN-2928: Lots of >>>>momentum, >>>> but clearly a work in progress. Two options here >>>>‹ If it is safe to ship it into 2.8 in a disable manner, we can >>>> get the early code into trunk and all the way int o2.8. >>>>‹ If it is not safe, it organically rolls over into 2.9 >>>> >>> >>> I'll review the changes on YARN-2928 to see what impact it has (if >>>any) if >>> the timeline service v.2 is disabled. >>> >>> Another condition for it to make 2.8 is whether the branch will be in a >>> shape in a couple of weeks such that it adds value for folks that want >>>to >>> test it. Hopefully it will become clearer soon. >>> >>> Sangjin >>> >
[jira] [Resolved] (HDFS-9370) TestDataNodeUGIProvider fails intermittently due to non-deterministic cache expiry.
[
https://issues.apache.org/jira/browse/HDFS-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-9370.
-
Resolution: Duplicate
> TestDataNodeUGIProvider fails intermittently due to non-deterministic cache
> expiry.
> ---
>
> Key: HDFS-9370
> URL: https://issues.apache.org/jira/browse/HDFS-9370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-9370.001.patch, HDFS-9370.002.patch
>
>
> {{TestDataNodeUGIProvider}} has hard-coded sleep times waiting for background
> expiration of entries in a Guava cache. I have seen this test suite fail
> intermittently, because expiration is not guaranteed to happen strictly on
> the boundary of the period defined by the cache's expiration time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HDFS-9370) TestDataNodeUGIProvider fails intermittently due to non-deterministic cache expiry.
[
https://issues.apache.org/jira/browse/HDFS-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth reopened HDFS-9370:
-
> TestDataNodeUGIProvider fails intermittently due to non-deterministic cache
> expiry.
> ---
>
> Key: HDFS-9370
> URL: https://issues.apache.org/jira/browse/HDFS-9370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-9370.001.patch, HDFS-9370.002.patch
>
>
> {{TestDataNodeUGIProvider}} has hard-coded sleep times waiting for background
> expiration of entries in a Guava cache. I have seen this test suite fail
> intermittently, because expiration is not guaranteed to happen strictly on
> the boundary of the period defined by the cache's expiration time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9459) hadoop-hdfs-native-client fails test build on Windows after transition to ctest.
Chris Nauroth created HDFS-9459:
---
Summary: hadoop-hdfs-native-client fails test build on Windows
after transition to ctest.
Key: HDFS-9459
URL: https://issues.apache.org/jira/browse/HDFS-9459
Project: Hadoop HDFS
Issue Type: Bug
Components: build, test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Blocker
HDFS-9369 transitioned to usage of {{ctest}} for running the HDFS native tests.
This broke the {{mvn test}} build on Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9458) TestBackupNode always binds to port 50070, which can cause bind failures.
Chris Nauroth created HDFS-9458:
---
Summary: TestBackupNode always binds to port 50070, which can
cause bind failures.
Key: HDFS-9458
URL: https://issues.apache.org/jira/browse/HDFS-9458
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
{{TestBackupNode}} does not override port settings to use a dynamically
selected port for the NameNode HTTP server. It uses the default of 50070
defined in hdfs-default.xml. This should be changed to select a dynamic port
to avoid bind errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9450) Fix failing HDFS tests on HDFS-7240 Ozone branch.
Chris Nauroth created HDFS-9450: --- Summary: Fix failing HDFS tests on HDFS-7240 Ozone branch. Key: HDFS-9450 URL: https://issues.apache.org/jira/browse/HDFS-9450 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Chris Nauroth Several test failures have been introduced on the HDFS-7240 Ozone feature branch. This issue tracks fixing those tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9443) Disabling HDFS client socket cache causes logging message printed to console for CLI commands.
Chris Nauroth created HDFS-9443:
---
Summary: Disabling HDFS client socket cache causes logging message
printed to console for CLI commands.
Key: HDFS-9443
URL: https://issues.apache.org/jira/browse/HDFS-9443
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Trivial
The HDFS client's socket cache can be disabled by setting
{{dfs.client.socketcache.capacity}} to {{0}}. When this is done, the
{{PeerCache}} class logs an info-level message stating that the cache is
disabled. This message is getting printed to the console for CLI commands,
which disrupts CLI output. This issue proposes to downgrade to debug-level
logging for this message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-190) DataNode should be marked as final to prevent subclassing
[
https://issues.apache.org/jira/browse/HDFS-190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-190.
Resolution: Won't Fix
We're now in a situation where the current codebase uses subclassing of
{{DataNode}} for some tests. There has been no activity on this issue for many
years, so it looks unlikely that it would be implemented. I'm closing it as
won't fix.
> DataNode should be marked as final to prevent subclassing
> -
>
> Key: HDFS-190
> URL: https://issues.apache.org/jira/browse/HDFS-190
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Steve Loughran
>Priority: Minor
>
> Reviewing the DataNode core, it starts a thread in its constructor calling
> back in to the Run() method. This is generally perceived as very dangerous,
> as if DataNode were ever subclassed, the subclass would start to be invoked
> in the run() method before its own constructor had finished working.
> 1. Consider splitting the constructor from the start() operation.
> 2. If this cannot be changed, mark DataNode as final so nobody can subclass
> it. Though if the latter were done, it would be convenient to have a method
> to let external management components poll for the health of the node, and to
> pick up reasons for the node shutting down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9409) DataNode shutdown does not guarantee full shutdown of all threads due to race condition.
Chris Nauroth created HDFS-9409:
---
Summary: DataNode shutdown does not guarantee full shutdown of all
threads due to race condition.
Key: HDFS-9409
URL: https://issues.apache.org/jira/browse/HDFS-9409
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Chris Nauroth
{{DataNode#shutdown}} is documented to return "only after shutdown is
complete". Even after completion of this method, it's possible that threads
started by the DataNode are still running. Race conditions in the shutdown
sequence may cause it to skip stopping and joining the {{BPServiceActor}}
threads.
This is likely not a big problem in normal operations, because these are daemon
threads that won't block overall process exit. It is more of a problem for
tests, because it makes it impossible to write reliable assertions that these
threads exited cleanly. For large test suites, it can also cause an
accumulation of unneeded threads, which might harm test performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-9404) Findbugs issue reported in BlockRecoveryWorker$RecoveryTaskContiguous.recover()
[ https://issues.apache.org/jira/browse/HDFS-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-9404. - Resolution: Duplicate Hi [~yzhangal]. This is tracked in HDFS-9401. Thanks! > Findbugs issue reported in > BlockRecoveryWorker$RecoveryTaskContiguous.recover() > --- > > Key: HDFS-9404 > URL: https://issues.apache.org/jira/browse/HDFS-9404 > Project: Hadoop HDFS > Issue Type: Bug > Components: HDFS >Reporter: Yongjun Zhang > > https://builds.apache.org/job/PreCommit-HDFS-Build/13431/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html > Reported: > Code Warning > ECCall to > org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration.equals(org.apache.hadoop.hdfs.protocol.DatanodeInfo) > in > org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.recover() > Details > EC_UNRELATED_TYPES: Call to equals() comparing different types > This method calls equals(Object) on two references of different class types > and analysis suggests they will be to objects of different classes at > runtime. Further, examination of the equals methods that would be invoked > suggest that either this call will always return false, or else the equals > method is not be symmetric (which is a property required by the contract for > equals in class Object). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9400) TestRollingUpgradeRollback fails on branch-2.
Chris Nauroth created HDFS-9400:
---
Summary: TestRollingUpgradeRollback fails on branch-2.
Key: HDFS-9400
URL: https://issues.apache.org/jira/browse/HDFS-9400
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Chris Nauroth
Priority: Blocker
During a Jenkins pre-commit run on branch-2 for the HDFS-9394 patch, we noticed
a pre-existing failure in {{TestRollingUpgradeRollback}}. I have confirmed
that this test is failing in branch-2 only. It passes in trunk, and it passes
in branch-2.7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: ReadOnly WebHDFS
Hello Laxman, I'm curious how this is a new problem after migration from HttpFs to WebHDFS. With the HttpFs deployment architecture, were you somehow proxying only the read-only operations? I would think this kind of thing could be achieved by writing a custom authentication filter, deploying that to the HDFS classpath, and then pointing to it by setting dfs.web.authentication.filter in hdfs-site.xml to the full name of that custom authentication filter class. The logic of the custom authentication filter would check for only read-only operations and reject the others. This is a solution that wouldn't require changes in WebHDFS itself. This is not a requirement I've heard from anyone else. I'm generally reluctant to add features without a widespread need. Still, if you want to file an HDFS JIRA for further discussion of your proposal, there is no harm in that. It might end up as a "Won't Fix", or perhaps others in the community see it differently from me, and we'd want to proceed. Thanks for sharing the work you've done! --Chris Nauroth On 11/6/15, 3:02 AM, "Laxman Ch" wrote: >Hi, > >We run a cluster with security set to simple. >Also, to some users, we had provided http access to HDFS via HttpFS >gateways. >However, this is not scaling and we are suffering from HttpFs gateway >choking problem. So, we wanted to enable WebHDFS directly on hadoop. But >this brings in the problem of security. Any user can simply delete >anything. And, we can't enable immediately enable kerberos security in >production. > >How about introducing a configuration to make WebHDFS readonly? >We patched this in our clusters cleanly and its working. > >Please revert with your comments if its a good idea to push this to >hadoop. >If yes, I will create a jira and submit patch. >-- >Thanks, >Laxman
[jira] [Created] (HDFS-9394) branch-2 hadoop-hdfs-client fails during FileSystem ServiceLoader initialization, because HftpFileSystem is missing.
Chris Nauroth created HDFS-9394:
---
Summary: branch-2 hadoop-hdfs-client fails during FileSystem
ServiceLoader initialization, because HftpFileSystem is missing.
Key: HDFS-9394
URL: https://issues.apache.org/jira/browse/HDFS-9394
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Reporter: Chris Nauroth
Priority: Critical
On branch-2, hadoop-hdfs-client contains a {{FileSystem}} service descriptor
that lists {{HftpFileSystem}} and {{HsftpFileSystem}}. These classes do not
reside in hadoop-hdfs-client. Instead, they reside in hadoop-hdfs. If the
application has hadoop-hdfs-client.jar on the classpath, but not
hadoop-hdfs.jar, then this can cause a {{ServiceConfigurationError}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9384) TestWebHdfsContentLength intermittently hangs and fails due to TCP conversation mismatch between client and server.
Chris Nauroth created HDFS-9384:
---
Summary: TestWebHdfsContentLength intermittently hangs and fails
due to TCP conversation mismatch between client and server.
Key: HDFS-9384
URL: https://issues.apache.org/jira/browse/HDFS-9384
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
{{TestWebHdfsContentLength}} runs a simple hand-coded HTTP server in a
background thread to simulate some WebHDFS server responses. In some
environments (notably Windows), I have observed that the test can hang and fail
intermittently. The root cause is that the server fails to fully consume the
client's input. This causes a mismatch in the TCP conversation state, and
ultimately the client side hangs, then aborts after the 60-second socket
timeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9378) hadoop-hdfs-client tests do not write logs.
Chris Nauroth created HDFS-9378: --- Summary: hadoop-hdfs-client tests do not write logs. Key: HDFS-9378 URL: https://issues.apache.org/jira/browse/HDFS-9378 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Minor The tests that have been split into the hadoop-hdfs-client module are not writing any log output, because there is no src/test/resources/log4j.properties file in the module. This makes it more difficult to troubleshoot test failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9370) TestDataNodeUGIProvider fails intermittently due to non-deterministic cache expiry.
Chris Nauroth created HDFS-9370:
---
Summary: TestDataNodeUGIProvider fails intermittently due to
non-deterministic cache expiry.
Key: HDFS-9370
URL: https://issues.apache.org/jira/browse/HDFS-9370
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
{{TestDataNodeUGIProvider}} has hard-coded sleep times waiting for background
expiration of entries in a Guava cache. I have seen this test suite fail
intermittently, because expiration is not guaranteed to happen strictly on the
boundary of the period defined by the cache's expiration time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9362) TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
Chris Nauroth created HDFS-9362:
---
Summary: TestAuditLogger#testAuditLoggerWithCallContext assumes
Unix line endings, fails on Windows.
Key: HDFS-9362
URL: https://issues.apache.org/jira/browse/HDFS-9362
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
{{TestAuditLogger#testAuditLoggerWithCallContext}} was added recently to
exercise the new audit logging with caller context functionality. The tests
assume Unix line endings by hard-coding "\n" in asserts. These tests fail on
Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.
Chris Nauroth created HDFS-9311: --- Summary: Support optional offload of NameNode HA service health checks to a separate RPC server. Key: HDFS-9311 URL: https://issues.apache.org/jira/browse/HDFS-9311 Project: Hadoop HDFS Issue Type: Bug Components: ha, namenode Reporter: Chris Nauroth Assignee: Chris Nauroth When a NameNode is overwhelmed with load, it can lead to resource exhaustion of the RPC handler pools (both client-facing and service-facing). Eventually, this blocks the health check RPC issued from ZKFC, which triggers a failover. Depending on fencing configuration, the former active NameNode may be killed. In an overloaded situation, the new active NameNode is likely to suffer the same fate, because client load patterns don't change after the failover. This can degenerate into flapping between the 2 NameNodes without real recovery. If a NameNode had been killed by fencing, then it would have to transition through safe mode, further delaying time to recovery. This issue proposes a separate, optional RPC server at the NameNode for isolating the HA health checks. These health checks are lightweight operations that do not suffer from contention issues on the namesystem lock or other shared resources. Isolating the RPC handlers is sufficient to avoid this situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9239) DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness
Chris Nauroth created HDFS-9239: --- Summary: DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness Key: HDFS-9239 URL: https://issues.apache.org/jira/browse/HDFS-9239 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: DataNode-Lifeline-Protocol.pdf This issue proposes introduction of a new feature: the DataNode Lifeline Protocol. This is an RPC protocol that is responsible for reporting liveness and basic health information about a DataNode to a NameNode. Compared to the existing heartbeat messages, it is lightweight and not prone to resource contention problems that can harm accurate tracking of DataNode liveness currently. The attached design document contains more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-9138) TestDatanodeStartupFixesLegacyStorageIDs fails on Windows due to failure to unpack old image tarball that contains hard links
[
https://issues.apache.org/jira/browse/HDFS-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-9138.
-
Resolution: Not A Problem
Yes, my earlier HDFS-8554 patch fixed this already. I was only seeing a test
failure on an older branch. I'm resolving this issue.
> TestDatanodeStartupFixesLegacyStorageIDs fails on Windows due to failure to
> unpack old image tarball that contains hard links
> -
>
> Key: HDFS-9138
> URL: https://issues.apache.org/jira/browse/HDFS-9138
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-9138.001.patch
>
>
> {{TestDatanodeStartupFixesLegacyStorageIDs#testUpgradeFrom22via26FixesStorageIDs}}
> uses a checked-in DataNode data directory that contains hard links. The
> hard links cannot be handled correctly by the commons-compress library used
> in the Windows implementation of {{FileUtil#unTar}}. The result is that the
> unpacked block files have 0 length, the block files reported to the NameNode
> are invalid, and therefore the mini-cluster never gets enough good blocks
> reported to leave safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9138) TestDatanodeStartupFixesLegacyStorageIDs fails on Windows due to failure to unpack old image tarball that contains hard links
Chris Nauroth created HDFS-9138:
---
Summary: TestDatanodeStartupFixesLegacyStorageIDs fails on Windows
due to failure to unpack old image tarball that contains hard links
Key: HDFS-9138
URL: https://issues.apache.org/jira/browse/HDFS-9138
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
{{TestDatanodeStartupFixesLegacyStorageIDs#testUpgradeFrom22via26FixesStorageIDs}}
uses a checked-in DataNode data directory that contains hard links. The hard
links cannot be handled correctly by the commons-compress library used in the
Windows implementation of {{FileUtil#unTar}}. The result is that the unpacked
block files have 0 length, the block files reported to the NameNode are
invalid, and therefore the mini-cluster never gets enough good blocks reported
to leave safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-9136) TestDFSUpgrade leaks file descriptors.
[
https://issues.apache.org/jira/browse/HDFS-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-9136.
-
Resolution: Not A Problem
I was mistaken. This actually got fixed as part of the test changes done in
HDFS-8846. I'm no longer seeing a problem after that patch. I'll resolve this.
> TestDFSUpgrade leaks file descriptors.
> --
>
> Key: HDFS-9136
> URL: https://issues.apache.org/jira/browse/HDFS-9136
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
>
> HDFS-8480 introduced code in {{TestDFSUpgrade#testPreserveEditLogs}} that
> opens edit log files and reads from them, but these files are never closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9136) TestDFSUpgrade leaks file descriptors.
Chris Nauroth created HDFS-9136:
---
Summary: TestDFSUpgrade leaks file descriptors.
Key: HDFS-9136
URL: https://issues.apache.org/jira/browse/HDFS-9136
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
HDFS-8480 introduced code in {{TestDFSUpgrade#testPreserveEditLogs}} that opens
edit log files and reads from them, but these files are never closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9128) TestWebHdfsFileContextMainOperations and TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on Windows.
Chris Nauroth created HDFS-9128: --- Summary: TestWebHdfsFileContextMainOperations and TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on Windows. Key: HDFS-9128 URL: https://issues.apache.org/jira/browse/HDFS-9128 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Trivial These tests do not override the default behavior of using the local file system test working directory to construct test paths. These paths will contain the ':' character on Windows due to the drive spec. HDFS rejects the ':' character as invalid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Even after HDFS-2856 JSVC References are require..?
I have no specific numbers to offer on this. Like you, I expect the impact is non-zero. I also expect the impact would depend on the setting of dfs.data.transfer.protection. For "authentication", it might just be a small up-front cost for the SASL authentication handshake. For "integrity" or "privacy", it becomes more a function of the payload (the block data transferred). For "privacy", I expect a similar hit to enabling data transfer encryption using dfs.encrypt.data.transfer, which is already known to be significant. I think it's a fair point to require a performance comparison before proceeding with any proposal of deprecating the jsvc deployment option. --Chris Nauroth On 9/14/15, 4:23 PM, "Colin P. McCabe" wrote: >Has anyone measured the overhead of running SASL on >DataTransferProtocol? I would expect it to be non-zero compared with >simply running on a low port. The CPU overhead especially could >regress performance on a typical Hadoop cluster. > >best, >Colin > >On Thu, Sep 10, 2015 at 9:55 AM, Chris Nauroth >wrote: >> Yes, I have a paragraph in the docs describing how someone would go >>about >> migrating a jsvc-based deployment to a SASL-based deployment. >> >> >>http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Se >>cu >> reMode.html#Secure_DataNode >> >> >> It's a non-trivial operation that starts by making sure everyone is on >>2.6 >> first. This includes client deployments, which are notoriously more >> difficult to control than server deployments. >> >> --Chris Nauroth >> >> >> >> >> On 9/10/15, 1:21 AM, "Steve Loughran" wrote: >> >>>SASL authenticates the DN on Hadoop 2.6+, but it requires the clients to >>>be using the 2.6+ JARs; you can't use it on the 2.2-2.5 artifacts. >>> >>>> On 9 Sep 2015, at 18:45, Allen Wittenauer wrote: >>>> >>>> >>>> FWIW, I still use and prefer jsvc, esp with the sudo trick in place. >>>> >>>> On Sep 9, 2015, at 9:35 AM, Chris Nauroth >>>>wrote: >>>> >>>>> AFAIK, the majority of existing deployments still use jsvc to run a >>>>> secured DataNode. It would be a backwards-incompatible change to >>>>>remove >>>>> support for this deployment model. For that reason, I would be -1 >>>>>for >>>>> removing jsvc support, at least in the 2.x line. >>>>> >>>>> >>>>> It's something that could be considered for 3.x if we think the >>>>>clean-up >>>>> benefit outweighs the incompatibility cost. Before we do that, I'd >>>>>prefer >>>>> to hear if end users are having success with the SASL deployment >>>>>model. >>>>> Brahma, are you asking because you run clusters with the SASL >>>>>approach? >>>>> If so, has it been working well? >>>>> >>>>> --Chris Nauroth >>>>> >>>>> >>>>> >>>>> >>>>> On 9/9/15, 9:25 AM, "Haohui Mai" wrote: >>>>> >>>>>> JSVC is no longer required. It causes a lot of headaches in >>>>>> deployments. It's definitely a good target for clean ups. >>>>>> >>>>>> ~Haohui >>>>>> >>>>>> On Wed, Sep 9, 2015 at 5:24 AM, Brahma Reddy Battula >>>>>> wrote: >>>>>>> Hi All, >>>>>>> >>>>>>> AFAIK JSVC added secure the block tokens(..?). >>>>>>> >>>>>>> Since block tokens are secure now (SASL used to secure the >>>>>>> DataTransferProtocol, which transfers file block content between >>>>>>>HDFS >>>>>>> clients and DataNodes),then can we remove jsvc now (script >>>>>>>files)..? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks & Regards >>>>>>> >>>>>>> Brahma Reddy Battula >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >> >
Re: kindly confirm the behavior if configs are disabled after setting
Hello Archana, If ACLs are disabled after ACLs have already been created in the namespace, then the behavior is: 1. All subsequent ACL-related API calls fail intentionally. It is impossible to create, update, or delete any ACLs. 2. The NameNode is still capable of loading existing ACL metadata stored in a pre-existing fsimage or edit log that was generated while ACLs were enabled. If ACLs are disabled, it would not cause the NameNode to abort while loading ACL-related metadata. 3. If ACLs are present on a file or directory from a prior configuration with ACLs enabled, then those ACLs are still enforced when users attempt to access the files. This behavior is by design. If you're interested in more of the background discussion that led to these design decisions, please see the comments in issues HDFS-5899 and HDFS-5925. --Chris Nauroth From: Archana talamani mailto:archana.talam...@huawei.com>> Reply-To: "hdfs-dev@hadoop.apache.org<mailto:hdfs-dev@hadoop.apache.org>" mailto:hdfs-dev@hadoop.apache.org>> Date: Friday, September 11, 2015 at 1:31 AM To: "hdfs-dev@hadoop.apache.org<mailto:hdfs-dev@hadoop.apache.org>" mailto:hdfs-dev@hadoop.apache.org>> Subject: kindly confirm the behavior if configs are disabled after setting Hi All, I have a query on ACL/Storage policy or in general any configs which provide enable/disable feature - Like, ACL is set on a file/directory, after which permission check will use ACLs for checking permission. What should be the behavior if ACLs are disabled after setting? whether it should use normal permissions or should continue using ACLs for access check? Kindly reply the expected behavior. Thanks, Archana Talamani HUAWEI TECHNOLOGIES INDIA PVT LTD. [cid:image001.png@01D0EC9A.60550E10] - This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
