Apache Hadoop qbt Report: trunk+JDK9 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/21/ [Apr 22, 2018 3:07:19 PM] (arp) HDFS-13055. Aggregate usage statistics from datanodes. Contributed by [Apr 23, 2018 2:49:35 AM] (inigoiri) HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs [Apr 23, 2018 8:02:09 AM] (sunilg) YARN-7956. [UI2] Avoid duplicating Components link under [Apr 23, 2018 8:06:27 AM] (sunilg) YARN-8177. Documentation changes for auto creation of Leaf Queues with [Apr 23, 2018 5:08:53 PM] (arp) HDFS-13433. webhdfs requests can be routed incorrectly in federated [Apr 23, 2018 6:13:18 PM] (cdouglas) HDFS-13408. MiniDFSCluster to support being built on randomized base [Apr 23, 2018 9:14:05 PM] (lei) Update CHANGES, RELEASENOTES, and jdiff for 3.0.2 release. [Apr 23, 2018 10:26:53 PM] (fabbri) HADOOP-14756 S3Guard: expose capability query in MetadataStore and add [Apr 23, 2018 10:44:15 PM] (rkanter) HADOOP-15390. Yarn RM logs flooded by DelegationTokenRenewer trying to [Apr 23, 2018 11:09:01 PM] (Bharat) HDFS-13356. Balancer:Set default value of minBlockSize to 10mb. [Apr 24, 2018 2:12:16 AM] (inigoiri) HDFS-13336. Test cases of TestWriteToReplica failed in windows. [Apr 24, 2018 3:25:23 AM] (yqlin) HDFS-13490. RBF: Fix setSafeMode in the Router. Contributed by Inigo [Apr 24, 2018 10:29:14 AM] (junping_du) YARN-7598. Document how to use classpath isolation for aux-services in [Apr 24, 2018 6:30:38 PM] (rkanter) MAPREDUCE-7072. mapred job -history prints duplicate counter in human [Apr 25, 2018 10:50:52 PM] (omalley) HDFS-8456. Introduce STORAGE_CONTAINER_SERVICE as a new NodeType. [Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8614. OzoneHandler : Add Quota Support. (Contributed by Anu [Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8448. Create REST Interface for Volumes. Contributed by Anu [Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8637. OzoneHandler : Add Error Table. (Contributed by Anu Engineer) [Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8634. OzoneHandler: Add userAuth Interface and Simple userAuth [Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8644. OzoneHandler : Add volume handler. (Contributed by Anu [Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8654. OzoneHandler : Add ACL support. (Contributed by Anu Engineer) [Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8680. OzoneHandler : Add Local StorageHandler support for volumes. [Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8717. OzoneHandler : Add common bucket objects. (Contributed by Anu [Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8753. Ozone: Unify StorageContainerConfiguration with [Apr 25, 2018 10:51:00 PM] (omalley) HDFS-8695. OzoneHandler : Add Bucket REST Interface. (aengineer) [Apr 25, 2018 10:51:36 PM] (omalley) HDFS-8527. OzoneHandler: Integration of REST interface and container [Apr 25, 2018 10:51:39 PM] (omalley) HDFS-8757 : OzoneHandler : Add localStorageHandler support for Buckets. [Apr 25, 2018 10:51:39 PM] (omalley) HDFS-9834. OzoneHandler : Enable MiniDFSCluster based testing for Ozone. [Apr 25, 2018 10:51:39 PM] (omalley) HDFS-9845. OzoneHandler : Support List and Info Volumes. Contributed by [Apr 25, 2018 10:51:51 PM] (omalley) HDFS-9853. Ozone: Add container definitions. Contributed by Anu [Apr 25, 2018 10:51:53 PM] (omalley) HDFS-9848. Ozone: Add Ozone Client lib for volume handling. Contributed [Apr 25, 2018 10:51:53 PM] (omalley) HDFS-9907. Exclude Ozone protobuf-generated classes from Findbugs [Apr 25, 2018 10:51:53 PM] (omalley) HDFS-9873. Ozone: Add container transport server. Contributed by Anu [Apr 25, 2018 10:51:58 PM] (omalley) HDFS-9891. Ozone: Add container transport client. Contributed by Anu [Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9920. Stop tracking CHANGES.txt in the HDFS-7240 feature branch. [Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9916. OzoneHandler : Add Key handler. Contributed by Anu Engineer. [Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9916. OzoneHandler : Add Key handler. Contributed by Anu Engineer. [Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9925. Ozone: Add Ozone Client lib for bucket handling. Contributed [Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9926. ozone : Add volume commands to CLI. Contributed by Anu [Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9944. Ozone : Add container dispatcher. Contributed by Anu [Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9961. Ozone: Add buckets commands to CLI. Contributed by Anu [Apr 25, 2018 10:52:00 PM] (omalley) HDFS-10180. Ozone: Refactor container Namespace. Contributed by Anu [Apr 25, 2018 10:52:00 PM] (omalley) HDFS-9960. OzoneHandler : Add localstorage support for keys. Contributed [Apr 25, 2018 10:52:07 PM] (omalley) HDFS-10179. Ozone: Adding logging support. Contributed by Anu Engineer. [Apr 25, 2018 10:52:09 PM] (omalley) HDFS-10196. Ozone : Enable better error reporting for failed commands in [Apr 25, 2018 10:52:25 PM] (omalley) HDFS-10202. ozone : Add key commands to CLI. Contributed by Anu [Apr 25, 2018 10:52:33 PM] (omalley
[jira] [Created] (HDFS-13515) NetUtils#connect should log remote address for NoRouteToHostException
Ted Yu created HDFS-13515: - Summary: NetUtils#connect should log remote address for NoRouteToHostException Key: HDFS-13515 URL: https://issues.apache.org/jira/browse/HDFS-13515 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ted Yu {code} hdfs.BlockReaderFactory: I/O error constructing remote block reader. java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2884) {code} In the above stack trace, the remote host was not logged. This makes troubleshooting a bit hard. NetUtils#connect should log remote address for NoRouteToHostException . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/765/ [Apr 27, 2018 2:38:15 AM] (yqlin) HDFS-13499. RBF: Show disabled name services in the UI. Contributed by [Apr 27, 2018 3:41:29 AM] (fabbri) HADOOP-13756 LocalMetadataStore#put(DirListingMetadata) should also put [Apr 27, 2018 4:56:57 AM] (sunilg) YARN-8205. Application State is not updated to ATS if AM launching is [Apr 27, 2018 5:03:42 AM] (sunilg) YARN-8215. ATS v2 returns invalid [Apr 27, 2018 5:12:30 AM] (sunilg) YARN-8004. Add unit tests for inter queue preemption for dominant [Apr 27, 2018 7:36:21 AM] (wwei) YARN-8152. Add chart in SLS to illustrate the throughput of the [Apr 27, 2018 10:21:37 AM] (aajisaka) HADOOP-14188. Remove the usage of [Apr 27, 2018 1:10:32 PM] (billie) YARN-8208. Add log statement for Docker client configuration file at [Apr 27, 2018 2:58:43 PM] (billie) YARN-8211. Yarn registry dns log finds BufferUnderflowException on port [Apr 27, 2018 4:24:43 PM] (eyang) YARN-8204. Added a flag to disable YARN service upgrade. [Apr 27, 2018 4:38:30 PM] (eyang) YARN-7781. Update YARN service documentation. Contributed by [Apr 27, 2018 5:05:55 PM] (weichiu) HADOOP-15382. Log kinit output in credential renewal thread. Contributed [Apr 27, 2018 5:28:10 PM] (rohithsharmaks) YARN-8221. RMWebServices also need to honor [Apr 27, 2018 8:07:38 PM] (wangda) YARN-8210. AMRMClient logging on every heartbeat to track updation of AM [Apr 27, 2018 8:08:43 PM] (wangda) YARN-8005. Add unit tests for queue priority with dominant resource [Apr 27, 2018 10:13:47 PM] (inigoiri) HDFS-13434. RBF: Fix dead links in RBF document. Contributed by Chetna [Apr 27, 2018 11:28:17 PM] (inigoiri) HDFS-13508. RBF: Normalize paths (automatically) when adding, updating, -1 overall The following subsystems voted -1: asflicense findbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-hdds/common Exceptional return value of java.io.File.mkdirs() ignored in org.apache.hadoop.utils.LevelDBStore.openDB(File, Options) At LevelDBStore.java:ignored in org.apache.hadoop.utils.LevelDBStore.openDB(File, Options) At LevelDBStore.java:[line 79] FindBugs : module:hadoop-hdds/container-service org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$ContainerBlocksDeletionACKProto.PARSER isn't final but should be At StorageContainerDatanodeProtocolProtos.java:be At StorageContainerDatanodeProtocolProtos.java:[line 17805] Class org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$ContainerBlocksDeletionACKProto defines non-transient non-serializable instance field unknownFields In StorageContainerDatanodeProtocolProtos.java:instance field unknownFields In StorageContainerDatanodeProtocolProtos.java org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$ContainerBlocksDeletionACKProto$DeleteBlockTransactionResult.PARSER isn't final but should be At StorageContainerDatanodeProtocolProtos.java:be At StorageContainerDatanodeProtocolProtos.java:[line 17928] Class org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$ContainerBlocksDeletionACKProto$DeleteBlockTransactionResult defines non-transient non-serializable instance field unknownFields In StorageContainerDatanodeProtocolProtos.java:instance field unknownFields In StorageContainerDatanodeProtocolProtos.java Useless control flow in org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$ContainerBlocksDeletionACKProto$DeleteBlockTransactionResult$Builder.maybeForceBuilderInitialization() At StorageContainerDatanodeProtocolProtos.java: At StorageContainerDatanodeProtocolProtos.java:[line 18177] org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$ContainerBlocksDeletionACKResponseProto.PARSER isn't final but should be At StorageContainerDatanodeProtocolProtos.java:be At StorageContainerDatanodeProtocolProtos.java:[line 17421] Class org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$ContainerBlocksDeletionACKResponseProto defines non-transient non-serializable instance field unknownFields In StorageContainerDatanodeProtocolProtos.java:instance field unknownFields In StorageContainerDatanodeProtocolProtos.java Useless control flow in org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$ContainerBlocksDeletionACKResponseProto$Builder.maybeForceBuilderInitialization() At StorageContainerDatanodeProtocolProtos.java: At StorageContainerDatanodeProtocolProtos.java:[line 17599] org.apache.hadoop.hdd
Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk
Sorry for landing late. Adding separate repo using "git submodule" wn't be good idea here, to get history/track commits for better maintaince..? Reference: http://blog.joncairns.com/2011/10/how-to-use-git-submodules/ On Fri, Apr 27, 2018 at 6:33 AM, Xiaoyu Yao wrote: > Thanks Owen for fixing this. Sorry, I was not able to spot the issue > before the merge is pushed up to trunk. > > I compared the changes and the results are the same. I also finish ozone > testing and verify the basic hdfs and ozone functionality are working fine. > Moving forward, I will add document on Hadoop wiki to avoid similar issue > in future. > > Regards, > Xiaoyu > > On 4/26/18, 9:59 AM, "Chris Douglas" wrote: > > Thank you for doing this, Owen. If the result is the same, then the > rebased version is much easier to work with. If not this, then we can > squash the branch into a single merge commit. The current state would > be a distant, third choice. > > +1 -C > > On Thu, Apr 26, 2018 at 5:49 AM, Owen O'Malley > wrote: > > Ok, it was bad enough that I spent yesterday afternoon hacking a fix > for it. > > > > I basically did a git rebase with always picking the current master > version > > and then put a new fixup commit to bring the branches into sync. > > > > The original - https://github.com/omalley/hadoop/tree/hdfs-7240 > > The rebase hack - https://github.com/omalley/ > hadoop/tree/hdfs-7240-rebase > > > > Compare the git log --graph difference for one of the recent commits: > > > > | * | | | | | | | | | | | | | | | | | | | | | | | commit > > 919ae746c3a4a424a77e6359af0abe3d7d2c1541 > > | | | | | | | | | | | | | | | | | | | | | | | | | Author: Weiwei > Yang < > > w...@apache.org> > > | | | | | | | | | | | | | | | | | | | | | | | | | Date: Wed Nov 1 > > 17:17:03 2017 +0800 > > | | | | | | | | | | | | | | | | | | | | | | | | | > > | | | | | | | | | | | | | | | | | | | | | | | | | HDFS-12750. > Ozone: > > Fix TestStorageContainerManager#testBlockDeletionTransactions. > Contributed > > by Xiaoyu Yao. > > | | | | | | | | | | | | | | | | | | | | | | | | | > > > > versus > > > > * commit 05592658078acef10c3995940510ff09792e > > | Author: Weiwei Yang > > | Date: Wed Nov 1 17:17:03 2017 +0800 > > | > > | HDFS-12750. Ozone: Fix > > TestStorageContainerManager#testBlockDeletionTransactions. > Contributed by > > Xiaoyu Yao. > > | > > > > I'd like permission to edit the hadoop git history to make the > change. > > > > Thoughts? > > > > ... Owen > > > > On Wed, Apr 25, 2018 at 11:47 AM, Xiaoyu Yao > wrote: > > > >> We followed the process laid out in a mail thread from 2015 that > had two > >> workflows proposed. > >> > >> “Git rebase” workflows and “Git merge” workflows. > >> Ozone had followed git merge workflow, which merged trunk at a > regular > >> frequency into ozone, and then ozone was merged back. > >> > >> Here is the mail that we followed for the merge process. > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac > >> > >> Thanks > >> Xiaoyu > >> > >> > >> > >> > >> > >> *From: *Owen O'Malley > >> *Date: *Wednesday, April 25, 2018 at 11:15 AM > >> *To: *Chris Douglas > >> *Cc: *Xiaoyu Yao , Jitendra Pandey < > >> jiten...@hortonworks.com>, Hdfs-dev > >> *Subject: *Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into > trunk > >> > >> > >> > >> *sigh* > >> > >> > >> > >> I asked you guys to rebase this before merge. It is tempting to > squash it > >> into a single commit. > >> > >> > >> > >> .. Owen > >> > >> > >> > >> On Wed, Apr 25, 2018 at 11:06 AM, Chris Douglas < > cdoug...@apache.org> > >> wrote: > >> > >> This really made a mess of trunk. Periodically merging trunk into > the > >> HDFS-7240 branch, then merging the whole thing back, created a > tangle > >> of references that's very difficult to work with (look at the output > >> of git log --graph). > >> > >> I'm not sure it's even possible to fix this, but this is why feature > >> branches should rebase and create a merge commit with --no-ff. -C > >> > >> > >> > >> On Tue, Apr 24, 2018 at 1:20 PM, Xiaoyu Yao > wrote: > >> > I just merged the branch to trunk (834 commits in total) > >> > Again, thanks for all who contributed to HDDS/Ozone! > >> > > >> > Cheers > >> > -Xiaoyu > >> > > >> > On 4/23/18, 7:26 PM, "Jitendra Pandey" > wrote: > >> > > >> > The vote passes with many +1s (12 committers + 5 > contributors) and > >> no -1. > >> > > >> > Thanks everyone for voting. > >> > > >> > On 4/17/18, 5:19 AM, "Jitendra Pandey" < > jiten...@hortonworks.com> > >> wrote: >