Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[Apache Jenkins Server]

For more details, see https://builds.apache.org/job/hadoop-trunk-win/429/

[Apr 5, 2018 9:28:36 PM] (sekikn) YETUS-622. docker-cleanup removes non-Yetus 
docker images even if not in
[Apr 5, 2018 9:29:47 PM] (sekikn) YETUS-623. docker-cleanup doesn't support 
options which start with
[Apr 5, 2018 4:59:10 PM] (lei) HDFS-13350. Negative legacy block ID will 
confuse Erasure Coding to be
[Apr 5, 2018 5:23:42 PM] (haibochen) YARN-6936. [Atsv2] Retrospect storing 
entities into sub application
[Apr 5, 2018 6:02:35 PM] (rohithsharmaks) YARN-8119. [UI2] Timeline Server 
address' url scheme should be removed
[Apr 5, 2018 7:00:52 PM] (weiy) HDFS-13353. RBF: 
TestRouterWebHDFSContractCreate failed. Contributed by
[Apr 5, 2018 10:52:39 PM] (wangda) Added CHANGES/RELEASES/Jdiff for 3.1.0 
release
[Apr 6, 2018 3:23:17 AM] (fabbri) HADOOP-14759 S3GuardTool prune to prune 
specific bucket entries.




-1 overall


The following subsystems voted -1:
compile mvninstall unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
unit


Specific tests:

Failed junit tests :

   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec 
   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend 
   hadoop.fs.TestFsShellCopy 
   hadoop.fs.TestFsShellList 
   hadoop.fs.TestLocalFileSystem 
   hadoop.fs.TestRawLocalFileSystemContract 
   hadoop.fs.TestSymlinkLocalFSFileContext 
   hadoop.fs.TestTrash 
   hadoop.http.TestHttpServer 
   hadoop.http.TestHttpServerLogs 
   hadoop.io.nativeio.TestNativeIO 
   hadoop.ipc.TestIPC 
   hadoop.ipc.TestSocketFactory 
   hadoop.metrics2.impl.TestStatsDMetrics 
   hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal 
   hadoop.security.TestSecurityUtil 
   hadoop.security.TestShellBasedUnixGroupsMapping 
   hadoop.security.token.TestDtUtilShell 
   hadoop.util.TestNativeCodeLoader 
   hadoop.fs.TestResolveHdfsSymlink 
   hadoop.hdfs.crypto.TestHdfsCryptoStreams 
   hadoop.hdfs.qjournal.client.TestQuorumJournalManager 
   hadoop.hdfs.qjournal.server.TestJournalNode 
   hadoop.hdfs.qjournal.server.TestJournalNodeSync 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages 
   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica 
   hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage 
   hadoop.hdfs.server.datanode.TestBlockRecovery 
   hadoop.hdfs.server.datanode.TestBlockScanner 
   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector 
   hadoop.hdfs.server.datanode.TestDataNodeMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.datanode.TestHSync 
   hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame 
   hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC 
   hadoop.hdfs.server.mover.TestMover 
   hadoop.hdfs.server.mover.TestStorageMover 
   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics 
   
hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots 
   hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport 
   hadoop.hdfs.server.namenode.TestAddBlock 
   hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands 
   hadoop.hdfs.server.namenode.TestCheckpoint 
   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate 
   hadoop.hdfs.server.namenode.TestEditLogRace 
   hadoop.hdfs.server.namenode.TestFileTruncate 
   hadoop.hdfs.server.namenode.TestFsck 
   hadoop.hdfs.server.namenode

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/187/

[Apr 6, 2018 1:41:58 PM] (mackrorysd) HDFS-13176. WebHdfs file path gets 
truncated when having semicolon
[Apr 6, 2018 4:47:25 PM] (haibochen) YARN-8107. Give an informative message 
when incorrect format is used in
[Apr 6, 2018 11:38:28 PM] (subru) YARN-8110. AMRMProxy recover should catch for 
all throwable to avoid




-1 overall


The following subsystems voted -1:
docker


Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13409) FilterFileSystem#setConf does not update the configuration

[Ahmed Eldawy (JIRA)]

Ahmed Eldawy created HDFS-13409:
---

 Summary: FilterFileSystem#setConf does not update the configuration
 Key: HDFS-13409
 URL: https://issues.apache.org/jira/browse/HDFS-13409
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ahmed Eldawy


FilterFileSystem keeps its own copy of Configuration which is separate from the 
underlying filtered file system. When you call FilterFileSystem#setConf, it 
updates its own private configuration which is not accessible. When you call 
FilterFileSystem#getConf, it returns the Configuration of the underlying 
FileSystem. Please look at the code snippet below.

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
FilterFileSystem ffs = new FilterFileSystem(fs);
Configuration conf2 = new Configuration();
conf2.set("foo", "bar");
ffs.setConf(conf2);
assertEquals("bar", ffs.getConf().get("foo")); // This assertion should pass 
but it fails

 

To solve this problem, we need to implement the following function in 
FilterFileSystem

@Override

public void setConf(Configuration conf) {
fs.setConf(conf);
}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC0)

[Yufei Gu]

Thanks Lei for working on this!

+1 (non-binding)

   - Downloaded the binary tarball and verified the checksum.
   - Started a pseudo cluster inside one docker container
   - Run Resource Manager with Fair Scheduler
   - Verified distributed shell
   - Verified mapreduce pi job
   - Sanity checked RM WebUI

Best,

Yufei

On Fri, Apr 6, 2018 at 11:16 AM, Lei Xu  wrote:

> Hi, All
>
> I've created release candidate RC-0 for Apache Hadoop 3.0.2.
>
> Please note: this is an amendment for Apache Hadoop 3.0.1 release to
> fix shaded jars in apache maven repository. The codebase of 3.0.2
> release is the same as 3.0.1.  New bug fixes will be included in
> Apache Hadoop 3.0.3 instead.
>
> The release page is:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>
> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC0/
>
> The git tag is release-3.0.2-RC0, and the latest commit is
> 5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee
>
> The maven artifacts are available at
> https://repository.apache.org/content/repositories/orgapachehadoop-1096/
>
> Please try the release, especially, *verify the maven artifacts*, and vote.
>
> The vote will run 5 days, ending 4/11/2018.
>
> Thanks for everyone who helped to spot the error and proposed fixes!
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Apache Hadoop 2.7.6 Release Plan

[Konstantin Shvachko]

Just heads up, working on the release candidate now.
It's been a while, I know. But we had some blockers.

Thanks,
--Konstantin

On Thu, Feb 22, 2018 at 5:59 PM, Konstantin Shvachko 
wrote:

> Planning to do a maintenance 2.7.6 release with close to 40 changes since
> the last release.
> If there are no objections I'll start preparations.
>
> Please let me know if there are blockers:
> https://issues.apache.org/jira/browse/HDFS-12641?filter=12343253
>
> Thanks,
> --Konstantin
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Thanks guys for the additional votes! I just sent out announcement email.

Best,
Wangda

On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵  wrote:

> Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> based on:
>
> - Verified signatures
>
> - Verified checksums for source and binary artifacts
>
> - Built from source
>
> - Deployed a single node cluster
>
> - Verified web UIs, include Namenode, RM, etc.
>
> * Tried shell commands of HDFS and YARN
>
> * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>
>
> Thanks,
>
>
> Junping
>
>
>
> Wangda Tan 于2018年3月30日 周五下午12:15写道：
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>

Re: Apache Hadoop 2.9.1 Release Plan

[Wei-Chiu Chuang]

Done.

Both HDFS-11915 and HDFS-13364 are in branch-2.9 and branch-2.9.1.

Thank you Sammi for driving 2.9.1 release

On Fri, Apr 6, 2018 at 10:14 AM, Wei-Chiu Chuang 
wrote:

>
>1.
>   Looks like:
>2. HDFS-13347  breaks
>compilation, which is later fixed byHDFS-13364
>. But HDFS-13364 is
>only available in 2.9.2 (branch-2.9)
>
>
>
>
>
>Before I cherrypick HDFS-11915, I will cherry pick HDFS-13364 first
>into branch-2.9.1.
>
>
> On Fri, Apr 6, 2018 at 9:46 AM, Chen, Sammi  wrote:
>
>> Hi Wei-Chiu,
>>
>> HDFS-11915 improves the data reliability when there is power failure.
>> I'm happy to include it in 2.9.1 release.
>>
>> Would you help to cherry-pick it to both branch-2.9 and branch-2.9.1?
>>
>>
>> Thanks,
>> Sammi
>>
>> -Original Message-
>> From: Wei-Chiu Chuang [mailto:weic...@cloudera.com]
>> Sent: Thursday, April 5, 2018 1:30 AM
>> To: Chen, Sammi 
>> Cc: hdfs-dev ; mapreduce-...@hadoop.apache.or
>> g; common-...@hadoop.apache.org; yarn-...@hadoop.apache.org
>> Subject: Re: Apache Hadoop 2.9.1 Release Plan
>>
>> Sorry Sammi I was late to this thread.
>> Please considering incorporating HDFS-11915. Sync rbw dir on the first
>> hsync() to avoid file lost on power failure.
>> I thought it was already in 2.9.1 but turns out it didn't land. The
>> cherry pick to branch-2.9 is conflict free.
>>
>> On Mon, Apr 2, 2018 at 4:34 AM, Chen, Sammi  wrote:
>>
>> > Hi All,
>> >
>> > Today I have created branch-2.9.1 from branch-2.9 and started creating
>> the
>> > RC0  based on branch-2.9.1.   But due to the corporate network
>> conditions
>> > and my not full privileges on Hadoop,   it will take a while for RC0 to
>> > come out.
>> >
>> > If you have anything want to commit to branch-2.9.1,  please let me
>> know.
>> >
>> > Also I will update fix version of all  2.9.1 JIRAs and moved all
>> > unresolved JIRA with target version = 2.9.1 to 2.9.2.
>> >
>> >
>> >
>> > Bests,
>> > Sammi Chen
>> >
>> > -Original Message-
>> > From: Chen, Sammi [mailto:sammi.c...@intel.com]
>> > Sent: Friday, March 30, 2018 3:55 PM
>> > To: hdfs-dev ;
>> > mapreduce-...@hadoop.apache.org; common-...@hadoop.apache.org;
>> > yarn-...@hadoop.apache.org
>> > Subject: Apache Hadoop 2.9.1 Release Plan
>> >
>> > Hi All,
>> >
>> > We have 47 changes on 2.9 branch since last release on Nov. 2017.
>>  There
>> > are 7 blockers, 5 critical issues and rest are normal bug fixes and
>> > feature improvements.
>> >
>> >
>> >
>> >
>> >
>> > Here are current tasks targeting for 2.9.1.  No critical and blockers
>> > so far.
>> >
>> > https://issues.apache.org/jira/issues/?jql=%22Target+
>> > Version%2Fs%22+%3D+2.9.1+AND+%28project+%3D+hadoop+OR+
>> > project+%3D+%22Hadoop+HDFS%22+OR+project+%3D+%22Hadoop+YARN%
>> > 22+OR+project+%3D+%22Hadoop+Map%2FReduce%22+OR+project+%
>> > 3D+%22Hadoop+Common%22%29+AND+status+%21%3D+resolved+ORDER+
>> > BY+priority+DESC
>> >
>> >
>> > I plan to cut the 2.9.1 branch today, and try to deliver the RC0  ASAP.
>> >  Please let me know if you have any objections or suggestions.
>> >
>> >
>> >
>> >
>> >
>> >
>> > Bests,
>> >
>> > Sammi
>> >
>> >
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> >
>> >
>>
>>
>> --
>> A very happy Clouderan
>>
>
>
>
> --
> A very happy Clouderan
>



-- 
A very happy Hadoop contributor

[jira] [Created] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory

[Xiao Liang (JIRA)]

Xiao Liang created HDFS-13408:
-

 Summary: MiniDFSCluster to support being built on randomized base 
directory
 Key: HDFS-13408
 URL: https://issues.apache.org/jira/browse/HDFS-13408
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Reporter: Xiao Liang
Assignee: Xiao Liang


Generated files of MiniDFSCluster during test are not properly cleaned in 
Windows, which fails all subsequent test cases using the same default directory 
(Windows does not allow other processes to delete them). By migrating to 
randomized base directories, the conflict of test path of test cases will be 
avoided, even if they are running at the same time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 3.0.2 (RC0)

[Lei Xu]

Hi, All

I've created release candidate RC-0 for Apache Hadoop 3.0.2.

Please note: this is an amendment for Apache Hadoop 3.0.1 release to
fix shaded jars in apache maven repository. The codebase of 3.0.2
release is the same as 3.0.1.  New bug fixes will be included in
Apache Hadoop 3.0.3 instead.

The release page is:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release

New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC0/

The git tag is release-3.0.2-RC0, and the latest commit is
5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee

The maven artifacts are available at
https://repository.apache.org/content/repositories/orgapachehadoop-1096/

Please try the release, especially, *verify the maven artifacts*, and vote.

The vote will run 5 days, ending 4/11/2018.

Thanks for everyone who helped to spot the error and proposed fixes!

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.9.1 Release Plan

[Wei-Chiu Chuang]

   1.
  Looks like:
   2. HDFS-13347  breaks
   compilation, which is later fixed byHDFS-13364
   . But HDFS-13364 is
   only available in 2.9.2 (branch-2.9)





   Before I cherrypick HDFS-11915, I will cherry pick HDFS-13364 first into
   branch-2.9.1.


On Fri, Apr 6, 2018 at 9:46 AM, Chen, Sammi  wrote:

> Hi Wei-Chiu,
>
> HDFS-11915 improves the data reliability when there is power failure.  I'm
> happy to include it in 2.9.1 release.
>
> Would you help to cherry-pick it to both branch-2.9 and branch-2.9.1?
>
>
> Thanks,
> Sammi
>
> -Original Message-
> From: Wei-Chiu Chuang [mailto:weic...@cloudera.com]
> Sent: Thursday, April 5, 2018 1:30 AM
> To: Chen, Sammi 
> Cc: hdfs-dev ; mapreduce-...@hadoop.apache.org;
> common-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.9.1 Release Plan
>
> Sorry Sammi I was late to this thread.
> Please considering incorporating HDFS-11915. Sync rbw dir on the first
> hsync() to avoid file lost on power failure.
> I thought it was already in 2.9.1 but turns out it didn't land. The cherry
> pick to branch-2.9 is conflict free.
>
> On Mon, Apr 2, 2018 at 4:34 AM, Chen, Sammi  wrote:
>
> > Hi All,
> >
> > Today I have created branch-2.9.1 from branch-2.9 and started creating
> the
> > RC0  based on branch-2.9.1.   But due to the corporate network conditions
> > and my not full privileges on Hadoop,   it will take a while for RC0 to
> > come out.
> >
> > If you have anything want to commit to branch-2.9.1,  please let me know.
> >
> > Also I will update fix version of all  2.9.1 JIRAs and moved all
> > unresolved JIRA with target version = 2.9.1 to 2.9.2.
> >
> >
> >
> > Bests,
> > Sammi Chen
> >
> > -Original Message-
> > From: Chen, Sammi [mailto:sammi.c...@intel.com]
> > Sent: Friday, March 30, 2018 3:55 PM
> > To: hdfs-dev ;
> > mapreduce-...@hadoop.apache.org; common-...@hadoop.apache.org;
> > yarn-...@hadoop.apache.org
> > Subject: Apache Hadoop 2.9.1 Release Plan
> >
> > Hi All,
> >
> > We have 47 changes on 2.9 branch since last release on Nov. 2017.   There
> > are 7 blockers, 5 critical issues and rest are normal bug fixes and
> > feature improvements.
> >
> >
> >
> >
> >
> > Here are current tasks targeting for 2.9.1.  No critical and blockers
> > so far.
> >
> > https://issues.apache.org/jira/issues/?jql=%22Target+
> > Version%2Fs%22+%3D+2.9.1+AND+%28project+%3D+hadoop+OR+
> > project+%3D+%22Hadoop+HDFS%22+OR+project+%3D+%22Hadoop+YARN%
> > 22+OR+project+%3D+%22Hadoop+Map%2FReduce%22+OR+project+%
> > 3D+%22Hadoop+Common%22%29+AND+status+%21%3D+resolved+ORDER+
> > BY+priority+DESC
> >
> >
> > I plan to cut the 2.9.1 branch today, and try to deliver the RC0  ASAP.
> >  Please let me know if you have any objections or suggestions.
> >
> >
> >
> >
> >
> >
> > Bests,
> >
> > Sammi
> >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
>
>
> --
> A very happy Clouderan
>



-- 
A very happy Clouderan

RE: Apache Hadoop 2.9.1 Release Plan

[Chen, Sammi]

Hi Wei-Chiu,

HDFS-11915 improves the data reliability when there is power failure.  I'm 
happy to include it in 2.9.1 release. 

Would you help to cherry-pick it to both branch-2.9 and branch-2.9.1? 


Thanks,
Sammi 

-Original Message-
From: Wei-Chiu Chuang [mailto:weic...@cloudera.com] 
Sent: Thursday, April 5, 2018 1:30 AM
To: Chen, Sammi 
Cc: hdfs-dev ; mapreduce-...@hadoop.apache.org; 
common-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.9.1 Release Plan

Sorry Sammi I was late to this thread.
Please considering incorporating HDFS-11915. Sync rbw dir on the first
hsync() to avoid file lost on power failure.
I thought it was already in 2.9.1 but turns out it didn't land. The cherry pick 
to branch-2.9 is conflict free.

On Mon, Apr 2, 2018 at 4:34 AM, Chen, Sammi  wrote:

> Hi All,
>
> Today I have created branch-2.9.1 from branch-2.9 and started creating the
> RC0  based on branch-2.9.1.   But due to the corporate network conditions
> and my not full privileges on Hadoop,   it will take a while for RC0 to
> come out.
>
> If you have anything want to commit to branch-2.9.1,  please let me know.
>
> Also I will update fix version of all  2.9.1 JIRAs and moved all 
> unresolved JIRA with target version = 2.9.1 to 2.9.2.
>
>
>
> Bests,
> Sammi Chen
>
> -Original Message-
> From: Chen, Sammi [mailto:sammi.c...@intel.com]
> Sent: Friday, March 30, 2018 3:55 PM
> To: hdfs-dev ; 
> mapreduce-...@hadoop.apache.org; common-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Apache Hadoop 2.9.1 Release Plan
>
> Hi All,
>
> We have 47 changes on 2.9 branch since last release on Nov. 2017.   There
> are 7 blockers, 5 critical issues and rest are normal bug fixes and 
> feature improvements.
>
>
>
>
>
> Here are current tasks targeting for 2.9.1.  No critical and blockers 
> so far.
>
> https://issues.apache.org/jira/issues/?jql=%22Target+
> Version%2Fs%22+%3D+2.9.1+AND+%28project+%3D+hadoop+OR+
> project+%3D+%22Hadoop+HDFS%22+OR+project+%3D+%22Hadoop+YARN%
> 22+OR+project+%3D+%22Hadoop+Map%2FReduce%22+OR+project+%
> 3D+%22Hadoop+Common%22%29+AND+status+%21%3D+resolved+ORDER+
> BY+priority+DESC
>
>
> I plan to cut the 2.9.1 branch today, and try to deliver the RC0  ASAP.
>  Please let me know if you have any objections or suggestions.
>
>
>
>
>
>
> Bests,
>
> Sammi
>
>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>


--
A very happy Clouderan

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/

[Apr 5, 2018 4:59:10 PM] (lei) HDFS-13350. Negative legacy block ID will 
confuse Erasure Coding to be
[Apr 5, 2018 5:23:42 PM] (haibochen) YARN-6936. [Atsv2] Retrospect storing 
entities into sub application
[Apr 5, 2018 6:02:35 PM] (rohithsharmaks) YARN-8119. [UI2] Timeline Server 
address' url scheme should be removed
[Apr 5, 2018 7:00:52 PM] (weiy) HDFS-13353. RBF: 
TestRouterWebHDFSContractCreate failed. Contributed by
[Apr 5, 2018 10:52:39 PM] (wangda) Added CHANGES/RELEASES/Jdiff for 3.1.0 
release




-1 overall


The following subsystems voted -1:
unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.federation.router.TestRouterSafemode 
   
hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/xml.txt
  [4.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [300K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [24K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [48K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/743/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]

Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[DISCUSS] Apache Hadoop 2.8.4 Release Plan

[俊平堵]

Hi my community friends,
With 3.0.1 and 3.1.0 release coming out successfully (thanks to the
great work from Eddy and Wangda), I am planning to do a maintenance release
for 2.8.4 in next few weeks. This release already includes 65 fixes - some
of them are outstanding issues. The reminding critial issues are listed in:
https://s.apache.org/SQce. I am going to move out some issues that had no
progress for a while and keep watching for the left, so please help me to
check non of them belongs to blockers. If there are any other outstanding
issues or work that you think it should be included in 2.8.4, please call
it out here. Thanks!

Thanks,

Junping

[jira] [Created] (HDFS-13407) Ozone: Use separated version schema for Hdds/Ozone projects

[Elek, Marton (JIRA)]

Elek, Marton created HDFS-13407:
---

 Summary: Ozone: Use separated version schema for Hdds/Ozone 
projects
 Key: HDFS-13407
 URL: https://issues.apache.org/jira/browse/HDFS-13407
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: HDFS-7240
Reporter: Elek, Marton
Assignee: Elek, Marton


The community is voted to manage Hdds/Ozone in-tree but with different release 
cycle. To achieve this we need to separated the versions of hdds/ozone projects 
from the mainline hadoop version (3.2.0-currently).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[俊平堵]

Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
based on:

- Verified signatures

- Verified checksums for source and binary artifacts

- Built from source

- Deployed a single node cluster

- Verified web UIs, include Namenode, RM, etc.

* Tried shell commands of HDFS and YARN

* Ran sample MR jobs, include PI, Sleep, Terasort, etc.


Thanks,


Junping



Wangda Tan 于2018年3月30日 周五下午12:15写道：

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>

[jira] [Resolved] (HDFS-13394) Ozone: ContainerID has incorrect package name

[Lokesh Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lokesh Jain resolved HDFS-13394.

Resolution: Not A Problem

> Ozone: ContainerID has incorrect package name
> -
>
> Key: HDFS-13394
> URL: https://issues.apache.org/jira/browse/HDFS-13394
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Nanda kumar
>Assignee: Lokesh Jain
>Priority: Major
>  Labels: newbie
>
> {{ContainerID}} package name and the directory structure where the class is 
> present doesn't match.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org