Re: [VOTE] Merging branch HDFS-7240 to trunk

[Sanjay Radia]

Owen,
Thanks for your proposal.
While I would have prefered to have HDSL in HDFS and also to be part
of Hadoop releases  for the reasons stated earlier in this thread,
I am willing to accept your proposal as a compromise to move this forward.

Jitendra, Anu, Daryn, Andrew, Konstantine your thoughts?

Thanks

Sanjay


On Mar 14, 2018, at 1:50 PM, Owen O'Malley 
> wrote:

This discussion seems to have died down coming closer consensus without a
resolution.

I'd like to propose the following compromise:

* HDSL become a subproject of Hadoop.
* HDSL will release separately from Hadoop. Hadoop releases will not
contain HDSL and vice versa.
* HDSL will get its own jira instance so that the release tags stay
separate.
* On trunk (as opposed to release branches) HDSL will be a separate module
in Hadoop's source tree. This will enable the HDSL to work on their trunk
and the Hadoop trunk without making releases for every change.
* Hadoop's trunk will only build HDSL if a non-default profile is enabled.
* When Hadoop creates a release branch, the RM will delete the HDSL module
from the branch.
* HDSL will have their own Yetus checks and won't cause failures in the
Hadoop patch check.

I think this accomplishes most of the goals of encouraging HDSL development
while minimizing the potential for disruption of HDFS development.

Thoughts? Andrew, Jitendra, & Sanjay?

Thanks,
  Owen

[VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Lei Xu]

Hi, all

I've created release candidate RC-1 for Apache Hadoop 3.0.1

Apache Hadoop 3.0.1 will be the first bug fix release for Apache
Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which
include 12
blockers and 17 are critical.

Please note:
* HDFS-12990. Change default NameNode RPC port back to 8020. It makes
incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
Hadoop 3.0.0 will be deprecated due to this change.

The release page is:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release

New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/

The git tag is release-3.0.1-RC1, and the latest commit is
496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0

The maven artifacts are available at:
https://repository.apache.org/content/repositories/orgapachehadoop-1081/

Please try the release and vote; the vote will run for the usual 5
days, ending on 3/22/2017 6pm PST time.

Thanks!

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: About reset branch-3.1 to trunk before release.

[Wangda Tan]

Thanks for sharing your thoughts.

We have done build and single node cluster deploy / test for the latest
trunk code (commit: 49c747ab187d0650143205ba57ca19607ec4c6bd). Since there
are no objections, so I will go ahead to do the branch replace.

Since we don't have force push permission to release branches. I just filed
https://issues.apache.org/jira/browse/INFRA-16204 to get help from Apache
infra team.

Please hold any commits to branch-3.1, will keep this email thread posted.

Best,
Wangda

On Wed, Mar 14, 2018 at 3:14 PM, Vinod Kumar Vavilapalli  wrote:

> I see one new feature: https://issues.apache.org/jira/browse/YARN-7626:
> Allow regular expression matching in container-executor.cfg for devices and
> named docker volumes mount.
>
> There are 21 sub-tasks. There are three feature-type JIRAs in those -
> https://issues.apache.org/jira/browse/YARN-7972,
> https://issues.apache.org/jira/browse/YARN-7891 and
> https://issues.apache.org/jira/browse/YARN-5015. These should be okay -
> not major disrupting features.
>
> Everything else is either a bug-fix or an improvement so we should be good.
>
> From the list, it doesn't look like resetting will destabilize 3.1, +1 for
> doing this.
>
> Thanks
> +Vinod
>
> > On Mar 14, 2018, at 1:54 PM, Wangda Tan  wrote:
> >
> > Hi mapreduce/yarn/common/hdfs-devs,
> >
> > As of now, we have all blockers done for 3.1.0 release [1]. The release
> is running behind schedule due to a few security-related issues. Because of
> this and since branch-3.1 is cut 5 weeks before on Feb 8, trunk 3.2 is
> already diverging. There're 64 commits in trunk but not in branch-3.1. [2]
> >
> > I took a quick scan of them, most of them are good fixes which we should
> bring to 3.1.0 as well. And this can also reduce differences between 3.2.0
> and 3.1.0 release for less maintenance burden in the future.
> >
> > Unless anyone objects, we will reset branch-3.1 to trunk in 1-2 days and
> cut RC after that.
> >
> > Thoughts?
> >
> > - Wangda
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.1.0 ORDER
> BY priority DESC
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0)
>
>

Apache Hadoop qbt Report: trunk+JDK9 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/15/

[Mar 12, 2018 2:01:42 PM] (stevel) HADOOP-15297. Make S3A etag => checksum 
feature optional. Contributed by
[Mar 12, 2018 4:58:56 PM] (inigoiri) HDFS-11399. Many tests fails in Windows 
due to injecting disk failures.
[Mar 12, 2018 7:07:22 PM] (aengineer) HDFS-10618. 
TestPendingReconstruction#testPendingAndInvalidate is flaky
[Mar 12, 2018 8:43:27 PM] (cdouglas) HADOOP-14742. Document multi-URI 
replication Inode for ViewFS.
[Mar 12, 2018 9:29:44 PM] (cdouglas) HDFS-12677. Extend 
TestReconstructStripedFile with a random EC policy.
[Mar 12, 2018 11:35:26 PM] (yufei) YARN-8024. LOG in class 
MaxRunningAppsEnforcer is initialized with a
[Mar 13, 2018 12:28:15 AM] (inigoiri) HDFS-13241. RBF: TestRouterSafemode 
failed if the port  is in use.
[Mar 13, 2018 2:15:51 AM] (yqlin) HDFS-10803.
[Mar 13, 2018 2:26:56 AM] (aajisaka) HDFS-12156. TestFSImage fails without 
-Pnative
[Mar 13, 2018 2:30:20 AM] (yqlin) HDFS-13253. RBF: Quota management incorrect 
parent-child relationship
[Mar 13, 2018 3:03:31 AM] (yqlin) HDFS-13226. RBF: Throw the exception if mount 
table entry validated
[Mar 13, 2018 3:05:39 AM] (cdouglas) HADOOP-14696. parallel tests don't work 
for Windows. Contributed by
[Mar 13, 2018 3:41:37 AM] (xyao) HDFS-13141. WebHDFS: Add support for getting 
snasphottable directory
[Mar 13, 2018 4:43:22 PM] (cdouglas) HDFS-13271. WebHDFS: Add constructor in 
SnapshottableDirectoryStatus
[Mar 13, 2018 4:58:03 PM] (xiao) HDFS-12505. Extend TestFileStatusWithECPolicy 
with a random EC policy.
[Mar 13, 2018 5:07:26 PM] (xiao) HDFS-13239. Fix non-empty dir warning message 
when setting default EC
[Mar 13, 2018 5:14:05 PM] (xiao) HDFS-12587. Use Parameterized tests in 
TestBlockInfoStriped and
[Mar 13, 2018 5:36:16 PM] (xiao) Revert "HDFS-12505. Extend 
TestFileStatusWithECPolicy with a random EC
[Mar 13, 2018 5:48:45 PM] (xiao) HDFS-12505. Extend TestFileStatusWithECPolicy 
with a random EC policy.
[Mar 13, 2018 6:08:11 PM] (cdouglas) HDFS-12780. Fix spelling mistake in 
DistCpUtils.java. Contributed by
[Mar 13, 2018 7:36:57 PM] (szegedim) YARN-5764. NUMA awareness support for 
launching containers. Contributed
[Mar 13, 2018 8:55:18 PM] (cdouglas) HADOOP-15311. HttpServer2 needs a way to 
configure the acceptor/selector
[Mar 13, 2018 11:35:51 PM] (arp) HDFS-13235. DiskBalancer: Update Documentation 
to add newly added
[Mar 13, 2018 11:39:17 PM] (arp) HDFS-336. dfsadmin -report should report 
number of blocks from datanode.
[Mar 14, 2018 12:55:17 AM] (wangda) YARN-5015. Support sliding window retry 
capability for container
[Mar 14, 2018 12:55:51 AM] (wangda) YARN-7657. Queue Mapping could provide 
options to provide 'user'
[Mar 14, 2018 3:01:07 AM] (inigoiri) HADOOP-15308. TestConfiguration fails on 
Windows because of paths.
[Mar 14, 2018 3:20:59 AM] (yqlin) HDFS-12773. RBF: Improve State Store FS 
implementation. Contributed by
[Mar 14, 2018 4:12:20 AM] (rohithsharmaks) YARN-8022. ResourceManager UI 
cluster/app/ page fails to render.
[Mar 14, 2018 4:38:07 AM] (aajisaka) HADOOP-15294. TestUGILoginFromKeytab fails 
on Java9
[Mar 14, 2018 6:39:53 AM] (sammi.chen) HDFS-11600. Refactor 
TestDFSStripedOutputStreamWithFailure test classes.
[Mar 14, 2018 4:14:26 PM] (xyao) HDFS-13249. Document webhdfs support for 
getting snapshottable directory
[Mar 14, 2018 4:21:57 PM] (eyang) YARN-7999. Added file cache initializer for 
Linux container-executor.   
[Mar 14, 2018 4:42:48 PM] (arp) HDFS-13257. Code cleanup: INode never throws 
QuotaExceededException.
[Mar 14, 2018 5:38:00 PM] (xyao) HDFS-13275. Adding log for 
BlockPoolManager#refreshNamenodes failures.
[Mar 14, 2018 6:26:58 PM] (xyao) HDFS-13251. Avoid using hard coded datanode 
data dirs in unit tests.
[Mar 14, 2018 8:05:24 PM] (jlowe) MAPREDUCE-7064. Flaky test
[Mar 14, 2018 8:14:36 PM] (inigoiri) HDFS-13198. RBF: RouterHeartbeatService 
throws out CachedStateStore
[Mar 14, 2018 8:36:53 PM] (wangda) Revert "HADOOP-13707. If kerberos is enabled 
while HTTP SPNEGO is not
[Mar 14, 2018 10:47:56 PM] (fabbri) HADOOP-15278 log s3a at info. Contributed 
by Steve Loughran.
[Mar 15, 2018 3:06:04 AM] (yqlin) HDFS-13261. Fix incorrect null value check. 
Contributed by Jianfei
[Mar 15, 2018 4:59:51 AM] (xiao) HDFS-13246. FileInputStream redundant closes 
in readReplicasFromCache.
[Mar 15, 2018 7:12:07 AM] (aajisaka) HADOOP-15305. Replace 
FileUtils.writeStringToFile(File, String) with
[Mar 15, 2018 5:14:35 PM] (inigoiri) HDFS-12723.
[Mar 15, 2018 5:18:44 PM] (xyao) HDFS-13251. Avoid using hard coded datanode 
data dirs in unit
[Mar 15, 2018 5:32:30 PM] (inigoiri) HDFS-13224. RBF: Resolvers to support 
mount points across multiple
[Mar 15, 2018 6:02:27 PM] (xyao) HDFS-13280. WebHDFS: Fix NPE in get 
snasphottable directory list call.
[Mar 15, 2018 6:05:14 PM] (stevel) HADOOP-15209. DistCp to eliminate needless 
deletion of files under
[Mar 15, 2018 8:26:01 PM] (wangda) MAPREDUCE-7047. 

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/

[Mar 16, 2018 3:17:16 AM] (xiao) HADOOP-15234. Throw meaningful message on null 
when initializing
[Mar 16, 2018 10:57:31 AM] (wwei) YARN-7636. Re-reservation count may overflow 
when cluster resource
[Mar 16, 2018 5:29:19 PM] (inigoiri) HDFS-12886. Ignore minReplication for 
block recovery. Contributed by
[Mar 16, 2018 11:25:33 PM] (yufei) YARN-8039. Clean up log dir configuration in




-1 overall


The following subsystems voted -1:
findbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

Failed junit tests :

   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.TestReconstructStripedFile 
   hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage 
   hadoop.yarn.server.TestDiskFailures 
   hadoop.yarn.sls.TestSLSRunner 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/whitespace-eol.txt
  [9.2M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/whitespace-tabs.txt
  [288K]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [304K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [48K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/723/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [12K]

Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org