Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

2018-04-26 Thread Apache Jenkins Server 
For more details, see https://builds.apache.org/job/hadoop-trunk-win/449/

[Apr 25, 2018 9:02:42 PM] (lei) HDFS-13468. Add erasure coding metrics into 
ReadStatistics. (Contributed
[Apr 26, 2018 5:09:37 AM] (wangda) HADOOP-15411. AuthenticationFilter should use
[Apr 26, 2018 5:10:18 AM] (wangda) YARN-8193. YARN RM hangs abruptly (stops 
allocating resources) when




-1 overall


The following subsystems voted -1:
compile mvninstall unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
unit


Specific tests:

Failed junit tests :

   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec 
   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend 
   hadoop.fs.TestFileUtil 
   hadoop.fs.TestFsShellCopy 
   hadoop.fs.TestFsShellList 
   hadoop.fs.TestLocalFileSystem 
   hadoop.fs.TestRawLocalFileSystemContract 
   hadoop.fs.TestSymlinkLocalFSFileContext 
   hadoop.fs.TestTrash 
   hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem 
   hadoop.http.TestHttpServer 
   hadoop.http.TestHttpServerLogs 
   hadoop.io.nativeio.TestNativeIO 
   hadoop.ipc.TestIPC 
   hadoop.ipc.TestSocketFactory 
   hadoop.metrics2.impl.TestStatsDMetrics 
   hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal 
   hadoop.security.TestSecurityUtil 
   hadoop.security.TestShellBasedUnixGroupsMapping 
   hadoop.security.token.TestDtUtilShell 
   hadoop.util.TestNativeCodeLoader 
   hadoop.util.TestNodeHealthScriptRunner 
   hadoop.fs.TestResolveHdfsSymlink 
   hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy 
   hadoop.hdfs.crypto.TestHdfsCryptoStreams 
   hadoop.hdfs.qjournal.client.TestQuorumJournalManager 
   hadoop.hdfs.qjournal.server.TestJournalNode 
   hadoop.hdfs.qjournal.server.TestJournalNodeSync 
   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages 
   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation 
   hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage 
   hadoop.hdfs.server.datanode.TestBlockRecovery 
   hadoop.hdfs.server.datanode.TestBlockScanner 
   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector 
   hadoop.hdfs.server.datanode.TestDataNodeMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.datanode.TestHSync 
   hadoop.hdfs.server.datanode.TestStorageReport 
   hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame 
   hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC 
   hadoop.hdfs.server.mover.TestMover 
   hadoop.hdfs.server.mover.TestStorageMover 
   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics 
   
hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots 
   hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport 
   hadoop.hdfs.server.namenode.TestAddBlock 
   hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands 
   hadoop.hdfs.server.namenode.TestCheckpoint 
   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate 
   hadoop.hdfs.server.namenode.TestEditLogRace 
   hadoop.hdfs.server.namenode.TestFileTruncate 
   hadoop.hdfs.server.namenode.TestFsck 
   hadoop.hdfs.server.namenode.TestFSImage 
   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot 
   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport 
   hadoop.hdfs.server.namenode.TestNameNodeMXBean 
   hadoop.hdfs.server.namenode.TestNestedEncryptionZones 
   hadoop.hdfs.server.namenode.TestQuotaByStorageType 
   hadoop.hdfs.server.namenode.TestReencryptionHandler

RE: [VOTE] Release Apache Hadoop 2.9.1 (RC0)

2018-04-26 Thread Takanobu Asanuma 
Thanks for working on this, Sammi!

+1 (non-binding)
   - Verified checksums
   - Succeeded "mvn clean package -Pdist,native -Dtar -DskipTests"
   - Started hadoop cluster with 1 master and 5 slaves
   - Run TeraGen/TeraSort
   - Verified some hdfs operations
   - Verified Web UI (NameNode, ResourceManager(classic and V2), JobHistory, 
Timeline)

Thanks,
-Takanobu

> -Original Message-
> From: Jinhu Wu [mailto:jinhu.wu@gmail.com]
> Sent: Friday, April 27, 2018 12:39 PM
> To: Gabor Bota 
> Cc: Chen, Sammi ; junping...@apache.org; Hadoop
> Common ; Rushabh Shah ;
> hdfs-dev ; mapreduce-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
> 
> Thanks Sammi for driving the release work!
> 
> +1 (non-binding)
> 
> based on following verification work:
> - built succeed from source on Mac OSX 10.13.4, java version "1.8.0_151"
> - run hadoop-aliyun tests successfully on cn-shanghai endpoint
> - deployed a one node cluster and verified PI job
> - verfied word-count job by using hadoop-aliyun as storage.
> 
> Thanks,
> jinhu
> 
> On Fri, Apr 27, 2018 at 12:45 AM, Gabor Bota 
> wrote:
> 
> >   Thanks for the work Sammi!
> >
> >   +1 (non-binding)
> >
> >-   checked out git tag release-2.9.1-RC0
> >-   S3A unit (mvn test) and integration (mvn verify) test run were
> >successful on us-west-2
> >-   built from source on Mac OS X 10.13.4, openjdk 1.8.0_144 (zulu)
> >-   deployed on a 3 node cluster
> >-   verified pi job, teragen, terasort and teravalidate
> >
> >
> >   Regards,
> >   Gabor Bota
> >
> > On Wed, Apr 25, 2018 at 7:12 AM, Chen, Sammi  wrote:
> >
> > >
> > > Paste the links here,
> > >
> > > The artifacts are available here:  https://dist.apache.org/repos/
> > > dist/dev/hadoop/2.9.1-RC0/
> > >
> > > The RC tag in git is release-2.9.1-RC0. Last git commit SHA is
> > > e30710aea4e6e55e69372929106cf119af06fd0e.
> > >
> > > The maven artifacts are available at:
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1
> > > 115/
> > >
> > > My public key is available from:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > >
> > > Bests,
> > > Sammi
> > > -Original Message-
> > > From: Chen, Sammi [mailto:sammi.c...@intel.com]
> > > Sent: Wednesday, April 25, 2018 12:02 PM
> > > To: junping...@apache.org
> > > Cc: Hadoop Common ; Rushabh Shah <
> > > rusha...@oath.com>; hdfs-dev ;
> > > mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> > > Subject: RE: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
> > >
> > >
> > > Thanks Jason Lowe for the quick investigation to find out that the
> > > test failures belong to the test only.
> > >
> > > Based on the current facts, I would like to continue calling the
> > > VOTE for
> > > 2.9.1 RC0,  and extend the vote deadline to end of this week 4/27.
> > >
> > >
> > > I will add following note to the final release notes,
> > >
> > > HADOOP-15385
> Test
> > > case failures in Hadoop-distcp project doesn’t impact the distcp
> > > function in 2.9.1
> > >
> > >
> > > Bests,
> > > Sammi
> > > From: 俊平堵 [mailto:junping...@apache.org]
> > > Sent: Tuesday, April 24, 2018 11:50 PM
> > > To: Chen, Sammi 
> > > Cc: Hadoop Common ; Rushabh Shah <
> > > rusha...@oath.com>; hdfs-dev ;
> > > mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> > > Subject: Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
> > >
> > > Thanks for reporting the issue, Rushabh! Actually, we found that
> > > these test failures belong to test issues but not production issue,
> > > so not
> > really
> > > a solid blocker for release. Anyway, I will let RM of 2.9.1 to
> > > decide if
> > to
> > > cancel RC or not for this test issue.
> > >
> > > Thanks,
> > >
> > > Junping
> > >
> > >
> > > Chen, Sammi >
> > 于2018年4月24日
> > > 周二下午7:50写道：
> > > Hi Rushabh,
> > >
> > > Thanks for reporting the issue.  I will upload a new RC candidate
> > > soon after the test failing issue is resolved.
> > >
> > >
> > > Bests,
> > > Sammi Chen
> > > From: Rushabh Shah
> > > [mailto:rusha...@oath.com]
> > > Sent: Friday, April 20, 2018 5:12 AM
> > > To: Chen, Sammi >
> > > Cc: Hadoop Common
> > >  > .
> > > apache.org>>; hdfs-dev  > hdfs-dev@hadoop.apache.org>>;
> mapreduce-...@hadoop.apache.org > > mapreduce-...@hadoop.apache.org>; yarn-...@hadoop.apache.org > > yarn-...@hadoop.apache.org>
> > > Subject: Re: [VOTE]

Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)

2018-04-26 Thread Jinhu Wu 
Thanks Sammi for driving the release work!

+1 (non-binding)

based on following verification work:
- built succeed from source on Mac OSX 10.13.4, java version "1.8.0_151"
- run hadoop-aliyun tests successfully on cn-shanghai endpoint
- deployed a one node cluster and verified PI job
- verfied word-count job by using hadoop-aliyun as storage.

Thanks,
jinhu

On Fri, Apr 27, 2018 at 12:45 AM, Gabor Bota 
wrote:

>   Thanks for the work Sammi!
>
>   +1 (non-binding)
>
>-   checked out git tag release-2.9.1-RC0
>-   S3A unit (mvn test) and integration (mvn verify) test run were
>successful on us-west-2
>-   built from source on Mac OS X 10.13.4, openjdk 1.8.0_144 (zulu)
>-   deployed on a 3 node cluster
>-   verified pi job, teragen, terasort and teravalidate
>
>
>   Regards,
>   Gabor Bota
>
> On Wed, Apr 25, 2018 at 7:12 AM, Chen, Sammi  wrote:
>
> >
> > Paste the links here,
> >
> > The artifacts are available here:  https://dist.apache.org/repos/
> > dist/dev/hadoop/2.9.1-RC0/
> >
> > The RC tag in git is release-2.9.1-RC0. Last git commit SHA is
> > e30710aea4e6e55e69372929106cf119af06fd0e.
> >
> > The maven artifacts are available at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1115/
> >
> > My public key is available from:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Bests,
> > Sammi
> > -Original Message-
> > From: Chen, Sammi [mailto:sammi.c...@intel.com]
> > Sent: Wednesday, April 25, 2018 12:02 PM
> > To: junping...@apache.org
> > Cc: Hadoop Common ; Rushabh Shah <
> > rusha...@oath.com>; hdfs-dev ;
> > mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> > Subject: RE: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
> >
> >
> > Thanks Jason Lowe for the quick investigation to find out that the test
> > failures belong to the test only.
> >
> > Based on the current facts, I would like to continue calling the VOTE for
> > 2.9.1 RC0,  and extend the vote deadline to end of this week 4/27.
> >
> >
> > I will add following note to the final release notes,
> >
> > HADOOP-15385   Test
> > case failures in Hadoop-distcp project doesn’t impact the distcp function
> > in 2.9.1
> >
> >
> > Bests,
> > Sammi
> > From: 俊平堵 [mailto:junping...@apache.org]
> > Sent: Tuesday, April 24, 2018 11:50 PM
> > To: Chen, Sammi 
> > Cc: Hadoop Common ; Rushabh Shah <
> > rusha...@oath.com>; hdfs-dev ;
> > mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> > Subject: Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
> >
> > Thanks for reporting the issue, Rushabh! Actually, we found that these
> > test failures belong to test issues but not production issue, so not
> really
> > a solid blocker for release. Anyway, I will let RM of 2.9.1 to decide if
> to
> > cancel RC or not for this test issue.
> >
> > Thanks,
> >
> > Junping
> >
> >
> > Chen, Sammi >
> 于2018年4月24日
> > 周二下午7:50写道：
> > Hi Rushabh,
> >
> > Thanks for reporting the issue.  I will upload a new RC candidate soon
> > after the test failing issue is resolved.
> >
> >
> > Bests,
> > Sammi Chen
> > From: Rushabh Shah [mailto:rusha...@oath.com]
> > Sent: Friday, April 20, 2018 5:12 AM
> > To: Chen, Sammi >
> > Cc: Hadoop Common  .
> > apache.org>>; hdfs-dev  hdfs-dev@hadoop.apache.org>>; mapreduce-...@hadoop.apache.org > mapreduce-...@hadoop.apache.org>; yarn-...@hadoop.apache.org > yarn-...@hadoop.apache.org>
> > Subject: Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
> >
> > Hi Chen,
> > I am so sorry to bring this up now but there are 16 tests failing in
> > hadoop-distcp project.
> > I have opened a ticket and cc'ed Junping since he is branch-2.8 committer
> > but I missed to ping you.
> >
> > IMHO we should fix the unit tests before we release but I would leave
> upto
> > other members to give their opinion.
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>

Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk

2018-04-26 Thread Xiaoyu Yao 
Thanks Owen for fixing this. Sorry, I was not able to spot the issue before the 
merge is pushed up to trunk. 

I compared the changes and the results are the same. I also finish ozone 
testing and verify the basic hdfs and ozone functionality are working fine. 
Moving forward, I will add document on Hadoop wiki to avoid similar issue in 
future.

Regards,
Xiaoyu 

On 4/26/18, 9:59 AM, "Chris Douglas"  wrote:

Thank you for doing this, Owen. If the result is the same, then the
rebased version is much easier to work with. If not this, then we can
squash the branch into a single merge commit. The current state would
be a distant, third choice.

+1 -C

On Thu, Apr 26, 2018 at 5:49 AM, Owen O'Malley  
wrote:
> Ok, it was bad enough that I spent yesterday afternoon hacking a fix for 
it.
>
> I basically did a git rebase with always picking the current master 
version
> and then put a new fixup commit to bring the branches into sync.
>
> The original - https://github.com/omalley/hadoop/tree/hdfs-7240
> The rebase hack - https://github.com/omalley/hadoop/tree/hdfs-7240-rebase
>
> Compare the git log --graph difference for one of the recent commits:
>
> | * | | | | | | | | | | | | | | | | | | | | | | | commit
> 919ae746c3a4a424a77e6359af0abe3d7d2c1541
> | | | | | | | | | | | | | | | | | | | | | | | | | Author: Weiwei Yang <
> w...@apache.org>
> | | | | | | | | | | | | | | | | | | | | | | | | | Date:   Wed Nov 1
> 17:17:03 2017 +0800
> | | | | | | | | | | | | | | | | | | | | | | | | |
> | | | | | | | | | | | | | | | | | | | | | | | | | HDFS-12750. Ozone:
> Fix TestStorageContainerManager#testBlockDeletionTransactions. Contributed
> by Xiaoyu Yao.
> | | | | | | | | | | | | | | | | | | | | | | | | |
>
> versus
>
> * commit 05592658078acef10c3995940510ff09792e
> | Author: Weiwei Yang 
> | Date:   Wed Nov 1 17:17:03 2017 +0800
> |
> | HDFS-12750. Ozone: Fix
> TestStorageContainerManager#testBlockDeletionTransactions. Contributed by
> Xiaoyu Yao.
> |
>
> I'd like permission to edit the hadoop git history to make the change.
>
> Thoughts?
>
> ... Owen
>
> On Wed, Apr 25, 2018 at 11:47 AM, Xiaoyu Yao  wrote:
>
>> We followed the process laid out in a mail thread from 2015 that had two
>> workflows proposed.
>>
>> “Git rebase” workflows and “Git merge” workflows.
>> Ozone had followed git merge workflow, which merged trunk at a regular
>> frequency into ozone, and then ozone was merged back.
>>
>> Here is the mail that we followed for the merge process.
>> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>
>> Thanks
>> Xiaoyu
>>
>>
>>
>>
>>
>> *From: *Owen O'Malley 
>> *Date: *Wednesday, April 25, 2018 at 11:15 AM
>> *To: *Chris Douglas 
>> *Cc: *Xiaoyu Yao , Jitendra Pandey <
>> jiten...@hortonworks.com>, Hdfs-dev 
>> *Subject: *Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk
>>
>>
>>
>> *sigh*
>>
>>
>>
>> I asked you guys to rebase this before merge. It is tempting to squash it
>> into a single commit.
>>
>>
>>
>> .. Owen
>>
>>
>>
>> On Wed, Apr 25, 2018 at 11:06 AM, Chris Douglas 
>> wrote:
>>
>> This really made a mess of trunk. Periodically merging trunk into the
>> HDFS-7240 branch, then merging the whole thing back, created a tangle
>> of references that's very difficult to work with (look at the output
>> of git log --graph).
>>
>> I'm not sure it's even possible to fix this, but this is why feature
>> branches should rebase and create a merge commit with --no-ff. -C
>>
>>
>>
>> On Tue, Apr 24, 2018 at 1:20 PM, Xiaoyu Yao  wrote:
>> > I just merged the branch to trunk (834 commits in total)
>> > Again, thanks for all who contributed to HDDS/Ozone!
>> >
>> > Cheers
>> > -Xiaoyu
>> >
>> > On 4/23/18, 7:26 PM, "Jitendra Pandey"  
wrote:
>> >
>> > The vote passes with many +1s (12 committers + 5 contributors) and
>> no -1.
>> >
>> > Thanks everyone for voting.
>> >
>> > On 4/17/18, 5:19 AM, "Jitendra Pandey" 

>> wrote:
>> >
>> > Hi All,
>> >
>> >The community unanimously voted (https://s.apache.org/
>> HDDSMergeResult) to adopt
>> > HDDS/Ozone as a sub-project of Hadoop, here is the formal
>> vote for code merge.
>> >
>> > Here is a quick summary of the

[NOTIFICATION] Hadoop trunk rebased

2018-04-26 Thread Owen O'Malley 
As we discussed in hdfs-dev@hadoop, I did a force push to Hadoop's trunk to
replace the Ozone merge with a rebase.

That means that you'll need to rebase your branches.

.. Owen

[jira] [Created] (HDFS-13509) Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows

2018-04-26 Thread Xiao Liang (JIRA) 
Xiao Liang created HDFS-13509:
-

 Summary: Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, 
and fix TestFileAppend failures on Windows
 Key: HDFS-13509
 URL: https://issues.apache.org/jira/browse/HDFS-13509
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Xiao Liang
Assignee: Xiao Liang


breakHardlinks() of ReplicaInfo(branch-2)/LocalReplica(trunk) replaces file 
while the source is still opened as input stream, which will fail and throw 
exception on Windows. It's the cause of  unit test case 
org.apache.hadoop.hdfs.TestFileAppend#testBreakHardlinksIfNeeded failure on 
Windows.

Other test cases of TestFileAppend fail randomly on Windows due to sharing the 
same test folder, and the solution is using randomized base dir of 
MiniDFSCluster via HDFS-13408



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk

2018-04-26 Thread Jitendra Pandey 
+1, Thanks Owen!

On 4/26/18, 9:58 AM, "Chris Douglas"  wrote:

Thank you for doing this, Owen. If the result is the same, then the
rebased version is much easier to work with. If not this, then we can
squash the branch into a single merge commit. The current state would
be a distant, third choice.

+1 -C

On Thu, Apr 26, 2018 at 5:49 AM, Owen O'Malley  
wrote:
> Ok, it was bad enough that I spent yesterday afternoon hacking a fix for 
it.
>
> I basically did a git rebase with always picking the current master 
version
> and then put a new fixup commit to bring the branches into sync.
>
> The original - https://github.com/omalley/hadoop/tree/hdfs-7240
> The rebase hack - https://github.com/omalley/hadoop/tree/hdfs-7240-rebase
>
> Compare the git log --graph difference for one of the recent commits:
>
> | * | | | | | | | | | | | | | | | | | | | | | | | commit
> 919ae746c3a4a424a77e6359af0abe3d7d2c1541
> | | | | | | | | | | | | | | | | | | | | | | | | | Author: Weiwei Yang <
> w...@apache.org>
> | | | | | | | | | | | | | | | | | | | | | | | | | Date:   Wed Nov 1
> 17:17:03 2017 +0800
> | | | | | | | | | | | | | | | | | | | | | | | | |
> | | | | | | | | | | | | | | | | | | | | | | | | | HDFS-12750. Ozone:
> Fix TestStorageContainerManager#testBlockDeletionTransactions. Contributed
> by Xiaoyu Yao.
> | | | | | | | | | | | | | | | | | | | | | | | | |
>
> versus
>
> * commit 05592658078acef10c3995940510ff09792e
> | Author: Weiwei Yang 
> | Date:   Wed Nov 1 17:17:03 2017 +0800
> |
> | HDFS-12750. Ozone: Fix
> TestStorageContainerManager#testBlockDeletionTransactions. Contributed by
> Xiaoyu Yao.
> |
>
> I'd like permission to edit the hadoop git history to make the change.
>
> Thoughts?
>
> ... Owen
>
> On Wed, Apr 25, 2018 at 11:47 AM, Xiaoyu Yao  wrote:
>
>> We followed the process laid out in a mail thread from 2015 that had two
>> workflows proposed.
>>
>> “Git rebase” workflows and “Git merge” workflows.
>> Ozone had followed git merge workflow, which merged trunk at a regular
>> frequency into ozone, and then ozone was merged back.
>>
>> Here is the mail that we followed for the merge process.
>> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>
>> Thanks
>> Xiaoyu
>>
>>
>>
>>
>>
>> *From: *Owen O'Malley 
>> *Date: *Wednesday, April 25, 2018 at 11:15 AM
>> *To: *Chris Douglas 
>> *Cc: *Xiaoyu Yao , Jitendra Pandey <
>> jiten...@hortonworks.com>, Hdfs-dev 
>> *Subject: *Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk
>>
>>
>>
>> *sigh*
>>
>>
>>
>> I asked you guys to rebase this before merge. It is tempting to squash it
>> into a single commit.
>>
>>
>>
>> .. Owen
>>
>>
>>
>> On Wed, Apr 25, 2018 at 11:06 AM, Chris Douglas 
>> wrote:
>>
>> This really made a mess of trunk. Periodically merging trunk into the
>> HDFS-7240 branch, then merging the whole thing back, created a tangle
>> of references that's very difficult to work with (look at the output
>> of git log --graph).
>>
>> I'm not sure it's even possible to fix this, but this is why feature
>> branches should rebase and create a merge commit with --no-ff. -C
>>
>>
>>
>> On Tue, Apr 24, 2018 at 1:20 PM, Xiaoyu Yao  wrote:
>> > I just merged the branch to trunk (834 commits in total)
>> > Again, thanks for all who contributed to HDDS/Ozone!
>> >
>> > Cheers
>> > -Xiaoyu
>> >
>> > On 4/23/18, 7:26 PM, "Jitendra Pandey"  
wrote:
>> >
>> > The vote passes with many +1s (12 committers + 5 contributors) and
>> no -1.
>> >
>> > Thanks everyone for voting.
>> >
>> > On 4/17/18, 5:19 AM, "Jitendra Pandey" 

>> wrote:
>> >
>> > Hi All,
>> >
>> >The community unanimously voted (https://s.apache.org/
>> HDDSMergeResult) to adopt
>> > HDDS/Ozone as a sub-project of Hadoop, here is the formal
>> vote for code merge.
>> >
>> > Here is a quick summary of the code changes:
>> >
>> > - As decided in the JIRA HDFS-10419, the project has been
>> renamed to Hadoop Distributed Data Store or HDDS.
>> > - HDDS becomes a sub-project of Hadoop.
>> > - Added a Maven profile that disables HDDS compilation by
>> default.
>> > - The releases

[jira] [Created] (HDFS-13508) Normalize paths (automatically) when adding, removing or listing mount table entries

2018-04-26 Thread Ekanth S (JIRA) 
Ekanth S created HDFS-13508:
---

 Summary: Normalize paths (automatically) when adding, removing or 
listing mount table entries
 Key: HDFS-13508
 URL: https://issues.apache.org/jira/browse/HDFS-13508
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Ekanth S


me@gateway-hawaii-all:/mnt/host/bin$ hdfs dfsrouteradmin -ls /home/move 
Mount Table Entries:
Source Destinations Owner Group Mode 
/home/move hdfs-oahu->/home/move me hadoop rwxr-xr-x

me@gateway-hawaii-all:/mnt/host/bin$ hdfs dfsrouteradmin -ls /home/move/
Mount Table Entries:
Source Destinations Owner Group Mode

me@gateway-hawaii-all:/mnt/host/bin$ hdfs dfsrouteradmin -rm /home/move/
Cannot remove mount point /home/move/

me@gateway-hawaii-all:/mnt/host/bin$ hdfs dfsrouteradmin -add /home/move/ 
hdfs-oahu /home/move/ -readonly
Cannot add mount point /home/move/

The slash '/' at the end should be normalized before calling the API from the 
CLI.

Note: add command fails with a terminating '/' . when it is an existing entry 
(it checks the not-normalized value with the normalized value in the 
mount-table). Adding a new mount point with '/' at the end works because the 
CLI normalizes the mount before calling the API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk

2018-04-26 Thread Chris Douglas 
Thank you for doing this, Owen. If the result is the same, then the
rebased version is much easier to work with. If not this, then we can
squash the branch into a single merge commit. The current state would
be a distant, third choice.

+1 -C

On Thu, Apr 26, 2018 at 5:49 AM, Owen O'Malley  wrote:
> Ok, it was bad enough that I spent yesterday afternoon hacking a fix for it.
>
> I basically did a git rebase with always picking the current master version
> and then put a new fixup commit to bring the branches into sync.
>
> The original - https://github.com/omalley/hadoop/tree/hdfs-7240
> The rebase hack - https://github.com/omalley/hadoop/tree/hdfs-7240-rebase
>
> Compare the git log --graph difference for one of the recent commits:
>
> | * | | | | | | | | | | | | | | | | | | | | | | | commit
> 919ae746c3a4a424a77e6359af0abe3d7d2c1541
> | | | | | | | | | | | | | | | | | | | | | | | | | Author: Weiwei Yang <
> w...@apache.org>
> | | | | | | | | | | | | | | | | | | | | | | | | | Date:   Wed Nov 1
> 17:17:03 2017 +0800
> | | | | | | | | | | | | | | | | | | | | | | | | |
> | | | | | | | | | | | | | | | | | | | | | | | | | HDFS-12750. Ozone:
> Fix TestStorageContainerManager#testBlockDeletionTransactions. Contributed
> by Xiaoyu Yao.
> | | | | | | | | | | | | | | | | | | | | | | | | |
>
> versus
>
> * commit 05592658078acef10c3995940510ff09792e
> | Author: Weiwei Yang 
> | Date:   Wed Nov 1 17:17:03 2017 +0800
> |
> | HDFS-12750. Ozone: Fix
> TestStorageContainerManager#testBlockDeletionTransactions. Contributed by
> Xiaoyu Yao.
> |
>
> I'd like permission to edit the hadoop git history to make the change.
>
> Thoughts?
>
> ... Owen
>
> On Wed, Apr 25, 2018 at 11:47 AM, Xiaoyu Yao  wrote:
>
>> We followed the process laid out in a mail thread from 2015 that had two
>> workflows proposed.
>>
>> “Git rebase” workflows and “Git merge” workflows.
>> Ozone had followed git merge workflow, which merged trunk at a regular
>> frequency into ozone, and then ozone was merged back.
>>
>> Here is the mail that we followed for the merge process.
>> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>
>> Thanks
>> Xiaoyu
>>
>>
>>
>>
>>
>> *From: *Owen O'Malley 
>> *Date: *Wednesday, April 25, 2018 at 11:15 AM
>> *To: *Chris Douglas 
>> *Cc: *Xiaoyu Yao , Jitendra Pandey <
>> jiten...@hortonworks.com>, Hdfs-dev 
>> *Subject: *Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk
>>
>>
>>
>> *sigh*
>>
>>
>>
>> I asked you guys to rebase this before merge. It is tempting to squash it
>> into a single commit.
>>
>>
>>
>> .. Owen
>>
>>
>>
>> On Wed, Apr 25, 2018 at 11:06 AM, Chris Douglas 
>> wrote:
>>
>> This really made a mess of trunk. Periodically merging trunk into the
>> HDFS-7240 branch, then merging the whole thing back, created a tangle
>> of references that's very difficult to work with (look at the output
>> of git log --graph).
>>
>> I'm not sure it's even possible to fix this, but this is why feature
>> branches should rebase and create a merge commit with --no-ff. -C
>>
>>
>>
>> On Tue, Apr 24, 2018 at 1:20 PM, Xiaoyu Yao  wrote:
>> > I just merged the branch to trunk (834 commits in total)
>> > Again, thanks for all who contributed to HDDS/Ozone!
>> >
>> > Cheers
>> > -Xiaoyu
>> >
>> > On 4/23/18, 7:26 PM, "Jitendra Pandey"  wrote:
>> >
>> > The vote passes with many +1s (12 committers + 5 contributors) and
>> no -1.
>> >
>> > Thanks everyone for voting.
>> >
>> > On 4/17/18, 5:19 AM, "Jitendra Pandey" 
>> wrote:
>> >
>> > Hi All,
>> >
>> >The community unanimously voted (https://s.apache.org/
>> HDDSMergeResult) to adopt
>> > HDDS/Ozone as a sub-project of Hadoop, here is the formal
>> vote for code merge.
>> >
>> > Here is a quick summary of the code changes:
>> >
>> > - As decided in the JIRA HDFS-10419, the project has been
>> renamed to Hadoop Distributed Data Store or HDDS.
>> > - HDDS becomes a sub-project of Hadoop.
>> > - Added a Maven profile that disables HDDS compilation by
>> default.
>> > - The releases of HDDS will be independent of Hadoop and
>> will have no impact to current Hadoop release process.
>> > - We have made HDDS a loadable module.
>> > - Cleaned up changes in HDFS/Hadoop Common to make sure HDDS
>> does not impact current users of HDFS.
>> >
>> > The vote will run for 7 days, I will start this vote with my
>> +1.
>> >
>> > Thanks
>> > Jitendra
>> >
>> >
>> > ---
>> --
>> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.
>> apache.org
>> >

Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)

2018-04-26 Thread Gabor Bota 
  Thanks for the work Sammi!

  +1 (non-binding)

   -   checked out git tag release-2.9.1-RC0
   -   S3A unit (mvn test) and integration (mvn verify) test run were
   successful on us-west-2
   -   built from source on Mac OS X 10.13.4, openjdk 1.8.0_144 (zulu)
   -   deployed on a 3 node cluster
   -   verified pi job, teragen, terasort and teravalidate


  Regards,
  Gabor Bota

On Wed, Apr 25, 2018 at 7:12 AM, Chen, Sammi  wrote:

>
> Paste the links here,
>
> The artifacts are available here:  https://dist.apache.org/repos/
> dist/dev/hadoop/2.9.1-RC0/
>
> The RC tag in git is release-2.9.1-RC0. Last git commit SHA is
> e30710aea4e6e55e69372929106cf119af06fd0e.
>
> The maven artifacts are available at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1115/
>
> My public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
>
> Bests,
> Sammi
> -Original Message-
> From: Chen, Sammi [mailto:sammi.c...@intel.com]
> Sent: Wednesday, April 25, 2018 12:02 PM
> To: junping...@apache.org
> Cc: Hadoop Common ; Rushabh Shah <
> rusha...@oath.com>; hdfs-dev ;
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: RE: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
>
>
> Thanks Jason Lowe for the quick investigation to find out that the test
> failures belong to the test only.
>
> Based on the current facts, I would like to continue calling the VOTE for
> 2.9.1 RC0,  and extend the vote deadline to end of this week 4/27.
>
>
> I will add following note to the final release notes,
>
> HADOOP-15385   Test
> case failures in Hadoop-distcp project doesn’t impact the distcp function
> in 2.9.1
>
>
> Bests,
> Sammi
> From: 俊平堵 [mailto:junping...@apache.org]
> Sent: Tuesday, April 24, 2018 11:50 PM
> To: Chen, Sammi 
> Cc: Hadoop Common ; Rushabh Shah <
> rusha...@oath.com>; hdfs-dev ;
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
>
> Thanks for reporting the issue, Rushabh! Actually, we found that these
> test failures belong to test issues but not production issue, so not really
> a solid blocker for release. Anyway, I will let RM of 2.9.1 to decide if to
> cancel RC or not for this test issue.
>
> Thanks,
>
> Junping
>
>
> Chen, Sammi >于2018年4月24日
> 周二下午7:50写道：
> Hi Rushabh,
>
> Thanks for reporting the issue.  I will upload a new RC candidate soon
> after the test failing issue is resolved.
>
>
> Bests,
> Sammi Chen
> From: Rushabh Shah [mailto:rusha...@oath.com]
> Sent: Friday, April 20, 2018 5:12 AM
> To: Chen, Sammi >
> Cc: Hadoop Common  apache.org>>; hdfs-dev >; mapreduce-...@hadoop.apache.org mapreduce-...@hadoop.apache.org>; yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)
>
> Hi Chen,
> I am so sorry to bring this up now but there are 16 tests failing in
> hadoop-distcp project.
> I have opened a ticket and cc'ed Junping since he is branch-2.8 committer
> but I missed to ping you.
>
> IMHO we should fix the unit tests before we release but I would leave upto
> other members to give their opinion.
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

[jira] [Created] (HDFS-13507) RBF: Remove update functionality from routeradmin's add cmd

2018-04-26 Thread Wei Yan (JIRA) 
Wei Yan created HDFS-13507:
--

 Summary: RBF: Remove update functionality from routeradmin's add 
cmd
 Key: HDFS-13507
 URL: https://issues.apache.org/jira/browse/HDFS-13507
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Wei Yan
Assignee: Gang Li


Follow up the discussion in HDFS-13326. We should remove the "update" 
functionality from routeradmin's add cmd, to make it consistent with RPC calls.

Note that: this is an incompatible change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk

2018-04-26 Thread Owen O'Malley 
Ok, it was bad enough that I spent yesterday afternoon hacking a fix for it.

I basically did a git rebase with always picking the current master version
and then put a new fixup commit to bring the branches into sync.

The original - https://github.com/omalley/hadoop/tree/hdfs-7240
The rebase hack - https://github.com/omalley/hadoop/tree/hdfs-7240-rebase

Compare the git log --graph difference for one of the recent commits:

| * | | | | | | | | | | | | | | | | | | | | | | | commit
919ae746c3a4a424a77e6359af0abe3d7d2c1541
| | | | | | | | | | | | | | | | | | | | | | | | | Author: Weiwei Yang <
w...@apache.org>
| | | | | | | | | | | | | | | | | | | | | | | | | Date:   Wed Nov 1
17:17:03 2017 +0800
| | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | HDFS-12750. Ozone:
Fix TestStorageContainerManager#testBlockDeletionTransactions. Contributed
by Xiaoyu Yao.
| | | | | | | | | | | | | | | | | | | | | | | | |

versus

* commit 05592658078acef10c3995940510ff09792e
| Author: Weiwei Yang 
| Date:   Wed Nov 1 17:17:03 2017 +0800
|
| HDFS-12750. Ozone: Fix
TestStorageContainerManager#testBlockDeletionTransactions. Contributed by
Xiaoyu Yao.
|

I'd like permission to edit the hadoop git history to make the change.

Thoughts?

... Owen

On Wed, Apr 25, 2018 at 11:47 AM, Xiaoyu Yao  wrote:

> We followed the process laid out in a mail thread from 2015 that had two
> workflows proposed.
>
> “Git rebase” workflows and “Git merge” workflows.
> Ozone had followed git merge workflow, which merged trunk at a regular
> frequency into ozone, and then ozone was merged back.
>
> Here is the mail that we followed for the merge process.
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>
> Thanks
> Xiaoyu
>
>
>
>
>
> *From: *Owen O'Malley 
> *Date: *Wednesday, April 25, 2018 at 11:15 AM
> *To: *Chris Douglas 
> *Cc: *Xiaoyu Yao , Jitendra Pandey <
> jiten...@hortonworks.com>, Hdfs-dev 
> *Subject: *Re: [RESULT][VOTE] Merge HDDS (HDFS-7240) *code* into trunk
>
>
>
> *sigh*
>
>
>
> I asked you guys to rebase this before merge. It is tempting to squash it
> into a single commit.
>
>
>
> .. Owen
>
>
>
> On Wed, Apr 25, 2018 at 11:06 AM, Chris Douglas 
> wrote:
>
> This really made a mess of trunk. Periodically merging trunk into the
> HDFS-7240 branch, then merging the whole thing back, created a tangle
> of references that's very difficult to work with (look at the output
> of git log --graph).
>
> I'm not sure it's even possible to fix this, but this is why feature
> branches should rebase and create a merge commit with --no-ff. -C
>
>
>
> On Tue, Apr 24, 2018 at 1:20 PM, Xiaoyu Yao  wrote:
> > I just merged the branch to trunk (834 commits in total)
> > Again, thanks for all who contributed to HDDS/Ozone!
> >
> > Cheers
> > -Xiaoyu
> >
> > On 4/23/18, 7:26 PM, "Jitendra Pandey"  wrote:
> >
> > The vote passes with many +1s (12 committers + 5 contributors) and
> no -1.
> >
> > Thanks everyone for voting.
> >
> > On 4/17/18, 5:19 AM, "Jitendra Pandey" 
> wrote:
> >
> > Hi All,
> >
> >The community unanimously voted (https://s.apache.org/
> HDDSMergeResult) to adopt
> > HDDS/Ozone as a sub-project of Hadoop, here is the formal
> vote for code merge.
> >
> > Here is a quick summary of the code changes:
> >
> > - As decided in the JIRA HDFS-10419, the project has been
> renamed to Hadoop Distributed Data Store or HDDS.
> > - HDDS becomes a sub-project of Hadoop.
> > - Added a Maven profile that disables HDDS compilation by
> default.
> > - The releases of HDDS will be independent of Hadoop and
> will have no impact to current Hadoop release process.
> > - We have made HDDS a loadable module.
> > - Cleaned up changes in HDFS/Hadoop Common to make sure HDDS
> does not impact current users of HDFS.
> >
> > The vote will run for 7 days, I will start this vote with my
> +1.
> >
> > Thanks
> > Jitendra
> >
> >
> > ---
> --
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.
> apache.org
> > For additional commands, e-mail:
> hdfs-dev-h...@hadoop.apache.org
> >
> >
> >
> >
> >
> >
> > ---
> --
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
> >
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands,

[jira] [Created] (HDFS-13506) RBF: Create destination directory when adding mount entry using router admin cmds.

2018-04-26 Thread Dibyendu Karmakar (JIRA) 
Dibyendu Karmakar created HDFS-13506:


 Summary: RBF: Create destination directory when adding mount entry 
using router admin cmds.
 Key: HDFS-13506
 URL: https://issues.apache.org/jira/browse/HDFS-13506
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Dibyendu Karmakar






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org