Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0
Thanks Brahma for voting and bringing this to my attention! On Thu, 19 Sep 2019 at 11:28, Brahma Reddy Battula wrote: > RohithThanks for driving the release > > +1 (Binding). > > --Built from the source > --Installed pseudo cluster > --Verified Basic hdfs shell command > --Ran Pi jobs > --Browsed the UI > > > *Rolling Upgrade:* > Following issue could have been merged.With out this, need to disable > token till rolling upgrade finalised. (Since one of main rolling upgrade > issue already merged (HDFS-13596)). > https://issues.apache.org/jira/browse/HDFS-14509 > This issue marked as blocker for 2.10 and still open!. Can anyone HDFS folks confirms this whether is is blocker for *hadoop-3.2.1* release? -Rohith Sharma K S
[jira] [Created] (HDDS-2152) Ozone client fails with OOM while writing a large (~300MB) key.
Aravindan Vijayan created HDDS-2152:
---
Summary: Ozone client fails with OOM while writing a large
(~300MB) key.
Key: HDDS-2152
URL: https://issues.apache.org/jira/browse/HDDS-2152
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Client
Reporter: Aravindan Vijayan
{code}
dd if=/dev/zero of=testfile bs=1024 count=307200
ozone sh key put /vol1/bucket1/key testfile
{code}
{code}
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at
java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) at
java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at
org.apache.hadoop.hdds.scm.storage.BufferPool.allocateBufferIfNeeded(BufferPool.java:66)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:234)
at
org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:129)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:211)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:193)
at
org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:96) at
org.apache.hadoop.ozone.web.ozShell.keys.PutKeyHandler.call(PutKeyHandler.java:117)
at
org.apache.hadoop.ozone.web.ozShell.keys.PutKeyHandler.call(PutKeyHandler.java:55)
at picocli.CommandLine.execute(CommandLine.java:1173) at
picocli.CommandLine.access$800(CommandLine.java:141)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2151) Ozone client prints the entire request payload in DEBUG level.
Aravindan Vijayan created HDDS-2151:
---
Summary: Ozone client prints the entire request payload in DEBUG
level.
Key: HDDS-2151
URL: https://issues.apache.org/jira/browse/HDDS-2151
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Aravindan Vijayan
In XceiverClientRatis.java:221, we have the following snippet where we have a
DEBUG line that prints out the entire Container Request proto.
{code}
ContainerCommandRequestProto finalPayload =
ContainerCommandRequestProto.newBuilder(request)
.setTraceID(TracingUtil.exportCurrentSpan())
.build();
boolean isReadOnlyRequest = HddsUtils.isReadOnly(finalPayload);
ByteString byteString = finalPayload.toByteString();
LOG.debug("sendCommandAsync {} {}", isReadOnlyRequest, finalPayload);
return isReadOnlyRequest ?
getClient().sendReadOnlyAsync(() -> byteString) :
getClient().sendAsync(() -> byteString);
{code}
This causes OOM while writing large (~300MB) keys.
{code}
SLF4J: Failed toString() invocation on an object of type
[org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto]
Reported exception:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649)
at java.lang.StringBuilder.append(StringBuilder.java:202)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:75)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormatEscaper.escapeBytes(TextFormatEscaper.java:94)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1836)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:436)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:449)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:376)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:338)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.print(TextFormat.java:325)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.print(TextFormat.java:68)
at
org.apache.ratis.thirdparty.com.google.protobuf.TextFormat.printToString(TextFormat.java:148)
at
org.apache.ratis.thirdparty.com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117)
at
org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
at
org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
at
org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
at
org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252)
at
org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:221)
at
org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:302)
at
org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:310)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:601)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:459)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:240)
at
org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:129)
SLF4J: Failed toString() invocation on an object of type
[org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos$ContainerCommandRequestProto]
Reported exception:
java.lang.OutOfMemoryError: Java heap space
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...
[jira] [Resolved] (HDDS-2021) Upgrade Guava library to v26 in hdds project
[ https://issues.apache.org/jira/browse/HDDS-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia resolved HDDS-2021. - Resolution: Not A Problem > Upgrade Guava library to v26 in hdds project > > > Key: HDDS-2021 > URL: https://issues.apache.org/jira/browse/HDDS-2021 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Affects Versions: 0.4.0 >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Upgrade Guava library to v26 in hdds project -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/449/ [Sep 18, 2019 5:31:33 PM] (ekrogen) HDFS-14569. Result of crypto -listZones is not formatted properly. [Sep 18, 2019 9:51:21 PM] (kihwal) HDFS-13959. TestUpgradeDomainBlockPlacementPolicy is flaky. Contributed [Sep 19, 2019 12:22:49 AM] (cliang) HDFS-14822. [SBN read] Revisit GlobalStateIdContext locking when getting - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2150) Update dependency versions to avoid security vulnerabilities
Hanisha Koneru created HDDS-2150: Summary: Update dependency versions to avoid security vulnerabilities Key: HDDS-2150 URL: https://issues.apache.org/jira/browse/HDDS-2150 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Hanisha Koneru Assignee: Hanisha Koneru The following dependency versions have known security vulnerabilities. We should update them to recent/ later versions. * Apache Thrift 0.11.0 * Apache Zookeeper 3.4.13 * Jetty Servlet 9.3.24 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2149) Replace findbugs with spotbugs
Dinesh Chitlangia created HDDS-2149: --- Summary: Replace findbugs with spotbugs Key: HDDS-2149 URL: https://issues.apache.org/jira/browse/HDDS-2149 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Dinesh Chitlangia Assignee: Dinesh Chitlangia Findbugs has been marked deprecated and all future work is now happening under SpotBugs project. This Jira is to investigate and possibly transition to Spotbugs in Ozone Ref1 - [https://mailman.cs.umd.edu/pipermail/findbugs-discuss/2017-September/004383.html] Ref2 - [https://spotbugs.github.io/] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree
+1. Thanks, Junping Elek, Marton 于2019年9月17日周二 下午5:48写道： > > > TLDR; I propose to move Ozone related code out from Hadoop trunk and > store it in a separated *Hadoop* git repository apache/hadoop-ozone.git > > > > > When Ozone was adopted as a new Hadoop subproject it was proposed[1] to > be part of the source tree but with separated release cadence, mainly > because it had the hadoop-trunk/SNAPSHOT as compile time dependency. > > During the last Ozone releases this dependency is removed to provide > more stable releases. Instead of using the latest trunk/SNAPSHOT build > from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now). > > As we have no more strict dependency between Hadoop trunk SNAPSHOT and > Ozone trunk I propose to separate the two code base from each other with > creating a new Hadoop git repository (apache/hadoop-ozone.git): > > With moving Ozone to a separated git repository: > > * It would be easier to contribute and understand the build (as of now > we always need `-f pom.ozone.xml` as a Maven parameter) > * It would be possible to adjust build process without breaking > Hadoop/Ozone builds. > * It would be possible to use different Readme/.asf.yaml/github > template for the Hadoop Ozone and core Hadoop. (For example the current > github template [2] has a link to the contribution guideline [3]. Ozone > has an extended version [4] from this guideline with additional > information.) > * Testing would be more safe as it won't be possible to change core > Hadoop and Hadoop Ozone in the same patch. > * It would be easier to cut branches for Hadoop releases (based on the > original consensus, Ozone should be removed from all the release > branches after creating relase branches from trunk) > > > What do you think? > > Thanks, > Marton > > [1]: > > https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E > [2]: > > https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md > [3]: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute > [4]: > > https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
[jira] [Resolved] (HDDS-2121) Create a shaded ozone filesystem (client) jar
[ https://issues.apache.org/jira/browse/HDDS-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-2121. -- Fix Version/s: 0.5.0 Resolution: Fixed > Create a shaded ozone filesystem (client) jar > - > > Key: HDDS-2121 > URL: https://issues.apache.org/jira/browse/HDDS-2121 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Arpit Agarwal >Assignee: Elek, Marton >Priority: Blocker > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > We need a shaded Ozonefs jar that does not include Hadoop ecosystem > components (Hadoop, HDFS, Ratis, Zookeeper). > A common expected use case for Ozone is Hadoop clients (3.2.0 and later) > wanting to access Ozone via the Ozone Filesystem interface. For these > clients, we want to add Ozone file system jar to the classpath, however we > want to use Hadoop ecosystem dependencies that are `provided` and already > expected to be in the client classpath. > Note that this is different from the legacy jar which bundles a shaded Hadoop > 3.2.0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14846) libhdfs tests are failing on trunk due to jni usage bugs
[
https://issues.apache.org/jira/browse/HDFS-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar resolved HDFS-14846.
-
Fix Version/s: 3.3.0
Resolution: Fixed
> libhdfs tests are failing on trunk due to jni usage bugs
>
>
> Key: HDFS-14846
> URL: https://issues.apache.org/jira/browse/HDFS-14846
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.3.0
>
>
> While working on HDFS-14564, I noticed that the libhdfs tests are failing on
> trunk (both on Hadoop QA and locally). I did some digging and found out that
> the {{-Xcheck:jni}} flag is causing a bunch of crashes. I haven't been able
> to pinpoint what caused this regression, but my best guess is that an upgrade
> in the JDK we use in Hadoop QA started causing these failures. I looked back
> at some old JIRAs and it looks like the tests work on Java 1.8.0_212, but
> Hadoop QA is running 1.8.0_222 (as is my local env) (I couldn't confirm this
> theory because I'm having trouble getting Java 1.8.0_212 installed next to
> 1.8.0_222 on my Ubuntu machine) (even after re-winding the commit history
> back to a known good commit where the libhdfs passed, the tests still fail,
> so I don't think a code change caused the regressions).
> The failures are a bunch of "FATAL ERROR in native method: Bad global or
> local ref passed to JNI" errors. After doing some debugging, it looks like
> {{-Xcheck:jni}} now errors out if any code tries to pass a local ref to
> {{DeleteLocalRef}} twice (previously it looked like it didn't complain) (we
> have some checks to avoid this, but it looks like they don't work as
> expected).
> There are a few places in the libhdfs code where this pattern causes a crash,
> as well as one place in {{JniBasedUnixGroupsMapping}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0
+1 (non-binding) - built from the source on centos7 - verified signatures & sha512 checksums - verified basic HDFS operations with CLI As Steve already mentioned, we must mention about Guava. Thanks for organizing the release Rohit. Thanks, Dinesh On Wed, Sep 18, 2019 at 2:20 PM Steve Loughran wrote: > On Wed, Sep 18, 2019 at 6:04 PM Rohith Sharma K S < > rohithsharm...@apache.org> > wrote: > > > Thanks Steve for detailed verification. Inline comment > > > > On Wed, 18 Sep 2019 at 20:34, Steve Loughran > > > wrote: > > > > > > > > > > +1 binding. > > > > > > > > One caveat: warn people that guava is now at 27.0 -and that if you > run > > > > with an older version of Guava things will inevitably break. > > > > > >: Could you please suggest what is the process to follow now If I > want > > to add into release notes? Should I withdraw RC0 and recreate RC1 with > > updated Release note in corresponding JIRA so that release script will > pick > > up that? Or any other way? > > > > > no, just remember to mention it to people. Though they'll find out soon > enough... >
[jira] [Created] (HDDS-2148) Remove redundant code in CreateBucketHandler.java
Dinesh Chitlangia created HDDS-2148:
---
Summary: Remove redundant code in CreateBucketHandler.java
Key: HDDS-2148
URL: https://issues.apache.org/jira/browse/HDDS-2148
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone CLI
Reporter: Dinesh Chitlangia
Assignee: Dinesh Chitlangia
{code:java}
if (isVerbose()) {
System.out.printf("Volume Name : %s%n", volumeName);
System.out.printf("Bucket Name : %s%n", bucketName);
if (bekName != null) {
bb.setBucketEncryptionKey(bekName);
System.out.printf("Bucket Encryption enabled with Key Name: %s%n",
bekName);
}
}
{code}
This jira aims to remove the redundant line
{{bb.setBucketEncryptionKey(bekName);}} as the same operation is performed in
the preceding code block. This code block is to print additional details if
verbose option was specified.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0
On Wed, Sep 18, 2019 at 6:04 PM Rohith Sharma K S wrote: > Thanks Steve for detailed verification. Inline comment > > On Wed, 18 Sep 2019 at 20:34, Steve Loughran > wrote: > > > > > > > +1 binding. > > > > > > One caveat: warn people that guava is now at 27.0 -and that if you run > > > with an older version of Guava things will inevitably break. > > > >: Could you please suggest what is the process to follow now If I want > to add into release notes? Should I withdraw RC0 and recreate RC1 with > updated Release note in corresponding JIRA so that release script will pick > up that? Or any other way? > no, just remember to mention it to people. Though they'll find out soon enough...
Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0
+1 (binding) Thanks Rohith the work with the release. * built from the source (archlinux) * verified signatures * verified sha512 checksums * started a docker-based pseudo cluster * tested basic HDFS operations with CLI * Checked if the sources are uploaded to the maven staging repo Note 1: I haven't seen the ./patchprocess/gpgagent.conf file in earlier releases and It seems to be included. But I don't think it's a blocker. Note 2: *.sha512 files can be improved before uploading with removing the absolute path (to make it easier to check with sha512sum -c). Marton On 9/18/19 4:25 PM, Ayush Saxena wrote: Thanks Rohith for driving the release. +1 (non binding) -Built from source on Ubuntu-18.04 -Successful Native build. -Verified basic HDFS Commands. -Verified basic Erasure Coding Commands. -Verified basic RBF commands. -Browsed HDFS UI. Thanks -Ayush On Wed, 18 Sep 2019 at 15:41, Weiwei Yang wrote: +1 (binding) Downloaded tarball, setup a pseudo cluster manually Verified basic HDFS operations, copy/view files Verified basic YARN operations, run sample DS jobs Verified basic YARN restful APIs, e.g cluster/nodes info etc Set and verified YARN node-attributes, including CLI Thanks Weiwei On Sep 18, 2019, 11:41 AM +0800, zhankun tang , wrote: +1 (non-binding). Installed and verified it by running several Spark job and DS jobs. BR, Zhankun On Wed, 18 Sep 2019 at 08:05, Naganarasimha Garla < naganarasimha...@apache.org> wrote: Verified the source and the binary tar and the sha512 checksums Installed and verified the basic hadoop operations (ran few MR tasks) +1. Thanks, + Naga On Wed, Sep 18, 2019 at 1:32 AM Anil Sadineni wrote: +1 (non-binding) On Tue, Sep 17, 2019 at 9:55 AM Santosh Marella wrote: +1 (non-binding) On Wed, Sep 11, 2019 at 12:26 AM Rohith Sharma K S < rohithsharm...@apache.org> wrote: Hi folks, I have put together a release candidate (RC0) for Apache Hadoop 3.2.1. The RC is available at: http://home.apache.org/~rohithsharmaks/hadoop-3.2.1-RC0/ The RC tag in git is release-3.2.1-RC0: https://github.com/apache/hadoop/tree/release-3.2.1-RC0 The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1226/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS This vote will run for 7 days(5 weekdays), ending on 18th Sept at 11:59 pm PST. I have done testing with a pseudo cluster and distributed shell job. My +1 to start. Thanks & Regards Rohith Sharma K S -- Thanks & Regards, Anil Sadineni Solutions Architect, Optlin Inc Ph: 571-438-1974 | www.optlin.com - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0
+0 * built from source taball. * launched 3-nodes cluster. * ran some examplr MR jobs. * ran some CLI file operations. Same as 3.1.3 RC0, verbose INFO message is emitted on data transfer operations.:: $ bin/hadoop fs -put README.txt '/c;d/' 2019-09-18 17:41:24,977 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false This was already fixed by HDFS-14759. It was cherry-picked to branch-3.2 and branch-3.1 yesterday. While this might not be worth for sinking RC0, I would not like to give bad impression about quality control of patch release. Thanks, Masatake Iwasaki On 9/11/19 16:26, Rohith Sharma K S wrote: Hi folks, I have put together a release candidate (RC0) for Apache Hadoop 3.2.1. The RC is available at: http://home.apache.org/~rohithsharmaks/hadoop-3.2.1-RC0/ The RC tag in git is release-3.2.1-RC0: https://github.com/apache/hadoop/tree/release-3.2.1-RC0 The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1226/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS This vote will run for 7 days(5 weekdays), ending on 18th Sept at 11:59 pm PST. I have done testing with a pseudo cluster and distributed shell job. My +1 to start. Thanks & Regards Rohith Sharma K S - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree
+1 > On Sep 17, 2019, at 2:49 AM, Elek, Marton wrote: > > > > TLDR; I propose to move Ozone related code out from Hadoop trunk and store it > in a separated *Hadoop* git repository apache/hadoop-ozone.git > > > > > When Ozone was adopted as a new Hadoop subproject it was proposed[1] to be > part of the source tree but with separated release cadence, mainly because it > had the hadoop-trunk/SNAPSHOT as compile time dependency. > > During the last Ozone releases this dependency is removed to provide more > stable releases. Instead of using the latest trunk/SNAPSHOT build from > Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now). > > As we have no more strict dependency between Hadoop trunk SNAPSHOT and Ozone > trunk I propose to separate the two code base from each other with creating a > new Hadoop git repository (apache/hadoop-ozone.git): > > With moving Ozone to a separated git repository: > > * It would be easier to contribute and understand the build (as of now we > always need `-f pom.ozone.xml` as a Maven parameter) > * It would be possible to adjust build process without breaking Hadoop/Ozone > builds. > * It would be possible to use different Readme/.asf.yaml/github template for > the Hadoop Ozone and core Hadoop. (For example the current github template > [2] has a link to the contribution guideline [3]. Ozone has an extended > version [4] from this guideline with additional information.) > * Testing would be more safe as it won't be possible to change core Hadoop > and Hadoop Ozone in the same patch. > * It would be easier to cut branches for Hadoop releases (based on the > original consensus, Ozone should be removed from all the release branches > after creating relase branches from trunk) > > > What do you think? > > Thanks, > Marton > > [1]: > https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E > [2]: > https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md > [3]: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute > [4]: > https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0
Thanks Steve for detailed verification. Inline comment On Wed, 18 Sep 2019 at 20:34, Steve Loughran wrote: > > > > +1 binding. > > > > One caveat: warn people that guava is now at 27.0 -and that if you run > > with an older version of Guava things will inevitably break. > >: Could you please suggest what is the process to follow now If I want to add into release notes? Should I withdraw RC0 and recreate RC1 with updated Release note in corresponding JIRA so that release script will pick up that? Or any other way?
[jira] [Created] (HDDS-2147) Include dumpstream in test report
Doroszlai, Attila created HDDS-2147:
---
Summary: Include dumpstream in test report
Key: HDDS-2147
URL: https://issues.apache.org/jira/browse/HDDS-2147
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: Tools
Reporter: Doroszlai, Attila
Assignee: Doroszlai, Attila
Include {{*.dumpstream}} in the unit test report, which may help finding out
the cause of {{Corrupted STDOUT}} warning of forked JVM.
{noformat:title=https://github.com/elek/ozone-ci/blob/5429d0982c3b13d311ec353dba198f2f5253757c/pr/pr-hdds-2141-4zm8s/unit/output.log#L333-L334}
[INFO] Running org.apache.hadoop.utils.TestMetadataStore
[WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
1. See FAQ web page and the dump file
/workdir/hadoop-hdds/common/target/surefire-reports/2019-09-18T12-58-05_531-jvmRun1.dumpstream
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2146) Optimize block write path performance by reducing no of watchForCommit calls
Shashikant Banerjee created HDDS-2146: - Summary: Optimize block write path performance by reducing no of watchForCommit calls Key: HDDS-2146 URL: https://issues.apache.org/jira/browse/HDDS-2146 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: 0.5.0 Currently, the watchForCommit calls from client to Ratis server for All replicated semantics happens when the max buffer limit is reached which can potentially be called 4 times as per the default configs for a single full block write. The idea here is inspect and add optimizations to reduce the no of watchForCommit calls. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2145) Optimize client read path by reading multiple chunks along with block info in a single rpc call.
Shashikant Banerjee created HDDS-2145: - Summary: Optimize client read path by reading multiple chunks along with block info in a single rpc call. Key: HDDS-2145 URL: https://issues.apache.org/jira/browse/HDDS-2145 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client, Ozone Datanode Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: 0.5.0 Currently, ozone client issues a getBlock call to read the metadata info from rocks Db on dn to get the chunkInfo and then chunk info is read one by one inn separate rpc calls in the read path. This can be optimized by piggybacking readChunk calls along with getBlock in a single rpc call to dn. This Jira aims to address this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree
> one thing to consider here as you are giving up your ability to make > changes in hadoop-* modules, including hadoop-common, and their > dependencies, in sync with your own code. That goes for filesystem contract > tests. > > are you happy with that? Yes. I think we can live with it. Fortunatelly the Hadoop parts which are used by Ozone (security + rpc) are stable enough, we didn't need bigger changes until now (small patches are already included in 3.1/3.2). I think it's better to use released Hadoop bits in Ozone anyway, and worst (best?) case we can try to do more frequent patch releases from Hadoop (if required). m. On 9/18/19 12:06 PM, Steve Loughran wrote: one thing to consider here as you are giving up your ability to make changes in hadoop-* modules, including hadoop-common, and their dependencies, in sync with your own code. That goes for filesystem contract tests. are you happy with that? On Tue, Sep 17, 2019 at 10:48 AM Elek, Marton wrote: TLDR; I propose to move Ozone related code out from Hadoop trunk and store it in a separated *Hadoop* git repository apache/hadoop-ozone.git When Ozone was adopted as a new Hadoop subproject it was proposed[1] to be part of the source tree but with separated release cadence, mainly because it had the hadoop-trunk/SNAPSHOT as compile time dependency. During the last Ozone releases this dependency is removed to provide more stable releases. Instead of using the latest trunk/SNAPSHOT build from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now). As we have no more strict dependency between Hadoop trunk SNAPSHOT and Ozone trunk I propose to separate the two code base from each other with creating a new Hadoop git repository (apache/hadoop-ozone.git): With moving Ozone to a separated git repository: * It would be easier to contribute and understand the build (as of now we always need `-f pom.ozone.xml` as a Maven parameter) * It would be possible to adjust build process without breaking Hadoop/Ozone builds. * It would be possible to use different Readme/.asf.yaml/github template for the Hadoop Ozone and core Hadoop. (For example the current github template [2] has a link to the contribution guideline [3]. Ozone has an extended version [4] from this guideline with additional information.) * Testing would be more safe as it won't be possible to change core Hadoop and Hadoop Ozone in the same patch. * It would be easier to cut branches for Hadoop releases (based on the original consensus, Ozone should be removed from all the release branches after creating relase branches from trunk) What do you think? Thanks, Marton [1]: https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E [2]: https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md [3]: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute [4]: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0
> > +1 binding. > > One caveat: warn people that guava is now at 27.0 -and that if you run > with an older version of Guava things will inevitably break. > > > steps to validate > == > > * downloaded src and binary artifacts > * after import of KEYS and trusting Rohith's key, validate GPG signatures > * test basic hadoop fs commands against s3a with s3guard and abfds > > > > > Validating S3A connector > > > * grabbed the latest build of my cloudstore diagnostics JAR > https://github.com/steveloughran/cloudstore/releases/tag/tag_2019-09-13 > * and set an env var to it: > set -gx CLOUDSTORE cloudstore/target/cloudstore-0.1-SNAPSHOT.jar > > bin/hadoop jar $CLOUDSTORE storediag s3a://hwdev-steve-ireland-new > > > > Diagnostics for filesystem s3a://hwdev-steve-ireland-new/ > = > > S3A FileSystem connector > ASF Filesystem Connector to Amazon S3 Storage and compatible stores > > https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html > > Hadoop information > == > > Hadoop 3.2.1 > Compiled by rohithsharmaks on 2019-09-10T15:56Z > Compiled with protoc 2.5.0 > From source with checksum 776eaf9eee9c0ffc370bcbc1888737 > > Required Classes > > > All these classes must be on the classpath > > class: org.apache.hadoop.fs.s3a.S3AFileSystem > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > class: com.amazonaws.services.s3.AmazonS3 > > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar > class: com.amazonaws.ClientConfiguration > > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar > > Optional Classes > > > These classes are needed in some versions of Hadoop. > And/or for optional features to work. > > class: com.amazonaws.services.dynamodbv2.AmazonDynamoDB > > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar > class: com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient > > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar > class: com.fasterxml.jackson.annotation.JacksonAnnotation > > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-annotations-2.9.8.jar > class: com.fasterxml.jackson.core.JsonParseException > > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-core-2.9.8.jar > class: com.fasterxml.jackson.databind.ObjectMapper > > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-databind-2.9.8.jar > class: org.joda.time.Interval > Not found on classpath: org.joda.time.Interval > class: org.apache.hadoop.fs.s3a.s3guard.S3Guard > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > class: org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > class: org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > class: org.apache.hadoop.fs.s3a.Invoker > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > class: org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > > then some classes which aren't in 3.2 and so which I didn't expect to > see. > > class: org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens > Not found on classpath: > org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens > class: com.amazonaws.services.s3.model.SelectObjectContentRequest > > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar > class: org.apache.hadoop.fs.s3a.select.SelectInputStream > Not found on classpath: > org.apache.hadoop.fs.s3a.select.SelectInputStream > class: org.apache.hadoop.fs.s3a.impl.RenameOperation > Not found on classpath: > org.apache.hadoop.fs.s3a.impl.RenameOperation > > > > + the command then executed basic list/read/write operations; all good. > > > > Validating abfs connector > = > > * set -gx HADOOP_OPTIONAL_TOOLS hadoop-azure > > > > > Diagnostics for filesystem abfs://contai...@someone.dfs.core.windows.net/ > > > > Azure Abfs connector > ASF Filesystem Connector to Microsoft Azure ABFS Storage > https://hadoop.apache.org/docs/current/hadoop-azure/index.html > > Hadoop information > == > > Hadoop 3.2.1 > Compiled by rohithsharmaks on 2019-09-10T15:56Z > Compiled with protoc 2.5.0 > From source with checksum 776eaf9eee9c0ffc370bcbc1888737 > > Environment Variables > = > >
Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0
Thanks Rohith for driving the release. +1 (non binding) -Built from source on Ubuntu-18.04 -Successful Native build. -Verified basic HDFS Commands. -Verified basic Erasure Coding Commands. -Verified basic RBF commands. -Browsed HDFS UI. Thanks -Ayush On Wed, 18 Sep 2019 at 15:41, Weiwei Yang wrote: > +1 (binding) > > Downloaded tarball, setup a pseudo cluster manually > Verified basic HDFS operations, copy/view files > Verified basic YARN operations, run sample DS jobs > Verified basic YARN restful APIs, e.g cluster/nodes info etc > Set and verified YARN node-attributes, including CLI > > Thanks > Weiwei > On Sep 18, 2019, 11:41 AM +0800, zhankun tang , > wrote: > > +1 (non-binding). > > Installed and verified it by running several Spark job and DS jobs. > > > > BR, > > Zhankun > > > > On Wed, 18 Sep 2019 at 08:05, Naganarasimha Garla < > > naganarasimha...@apache.org> wrote: > > > > > Verified the source and the binary tar and the sha512 checksums > > > Installed and verified the basic hadoop operations (ran few MR tasks) > > > > > > +1. > > > > > > Thanks, > > > + Naga > > > > > > On Wed, Sep 18, 2019 at 1:32 AM Anil Sadineni > > > wrote: > > > > > > > +1 (non-binding) > > > > > > > > On Tue, Sep 17, 2019 at 9:55 AM Santosh Marella > > > wrote: > > > > > > > > > +1 (non-binding) > > > > > > > > > > On Wed, Sep 11, 2019 at 12:26 AM Rohith Sharma K S < > > > > > rohithsharm...@apache.org> wrote: > > > > > > > > > > > Hi folks, > > > > > > > > > > > > I have put together a release candidate (RC0) for Apache Hadoop > > > 3.2.1. > > > > > > > > > > > > The RC is available at: > > > > > > http://home.apache.org/~rohithsharmaks/hadoop-3.2.1-RC0/ > > > > > > > > > > > > The RC tag in git is release-3.2.1-RC0: > > > > > > https://github.com/apache/hadoop/tree/release-3.2.1-RC0 > > > > > > > > > > > > > > > > > > The maven artifacts are staged at > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1226/ > > > > > > > > > > > > You can find my public key at: > > > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > > > > > This vote will run for 7 days(5 weekdays), ending on 18th Sept at > > > 11:59 > > > > > pm > > > > > > PST. > > > > > > > > > > > > I have done testing with a pseudo cluster and distributed shell > job. > > > My > > > > > +1 > > > > > > to start. > > > > > > > > > > > > Thanks & Regards > > > > > > Rohith Sharma K S > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Thanks & Regards, > > > > Anil Sadineni > > > > Solutions Architect, Optlin Inc > > > > Ph: 571-438-1974 | www.optlin.com > > > > > > > >
[jira] [Created] (HDFS-14854) Create improved decommission monitor implementation
Stephen O'Donnell created HDFS-14854: Summary: Create improved decommission monitor implementation Key: HDFS-14854 URL: https://issues.apache.org/jira/browse/HDFS-14854 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.3.0 Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell In HDFS-13157, we discovered a series of problems with the current decommission monitor implementation, such as: * Blocks are replicated sequentially disk by disk and node by node, and hence the load is not spread well across the cluster * Adding a node for decommission can cause the namenode write lock to be held for a long time. * Decommissioning nodes floods the replication queue and under replicated blocks from a future node or disk failure may way for a long time before they are replicated. * Blocks pending replication are checked many times under a write lock before they are sufficiently replicate, wasting resources In this Jira I propose to create a new implementation of the decommission monitor that resolves these issues. As it will be difficult to prove one implementation is better than another, the new implementation can be enabled or disabled giving the option of the existing implementation or the new one. I will attach a pdf with some more details on the design and then a version 1 patch shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14853) NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is deleted
Ranith Sardar created HDFS-14853:
Summary: NPE in DFSNetworkTopology#chooseRandomWithStorageType()
when the excludedNode is deleted
Key: HDFS-14853
URL: https://issues.apache.org/jira/browse/HDFS-14853
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ranith Sardar
Assignee: Ranith Sardar
{{org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.net.DFSNetworkTopology.chooseRandomWithStorageType(DFSNetworkTopology.java:229)
at
org.apache.hadoop.hdfs.net.DFSNetworkTopology.chooseRandomWithStorageType(DFSNetworkTopology.java:77)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14852) Remove of LowRedundancyBlocks do NOT remove the block from all queues
Fei Hui created HDFS-14852:
--
Summary: Remove of LowRedundancyBlocks do NOT remove the block
from all queues
Key: HDFS-14852
URL: https://issues.apache.org/jira/browse/HDFS-14852
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 3.1.2, 3.0.3, 3.2.0, 3.3.0
Reporter: Fei Hui
Assignee: Fei Hui
LowRedundancyBlocks.java
{code:java}
// Some comments here
if(priLevel >= 0 && priLevel < LEVEL
&& priorityQueues.get(priLevel).remove(block)) {
NameNode.blockStateChangeLog.debug(
"BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block {}"
+ " from priority queue {}",
block, priLevel);
decrementBlockStat(block, priLevel, oldExpectedReplicas);
return true;
} else {
// Try to remove the block from all queues if the block was
// not found in the queue for the given priority level.
for (int i = 0; i < LEVEL; i++) {
if (i != priLevel && priorityQueues.get(i).remove(block)) {
NameNode.blockStateChangeLog.debug(
"BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block" +
" {} from priority queue {}", block, i);
decrementBlockStat(block, i, oldExpectedReplicas);
return true;
}
}
}
return false;
}
{code}
Source code is above, the comments as follow
{quote}
// Try to remove the block from all queues if the block was
// not found in the queue for the given priority level.
{quote}
The function "remove" does NOT remove the block from all queues.
Function add from LowRedundancyBlocks.java is used on some places and maybe one
block in two or more queues.
We found that corrupt blocks mismatch corrupt files on NN web UI. Maybe it is
related to this.
Upload initial patch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0
+1 (binding) Downloaded tarball, setup a pseudo cluster manually Verified basic HDFS operations, copy/view files Verified basic YARN operations, run sample DS jobs Verified basic YARN restful APIs, e.g cluster/nodes info etc Set and verified YARN node-attributes, including CLI Thanks Weiwei On Sep 18, 2019, 11:41 AM +0800, zhankun tang , wrote: > +1 (non-binding). > Installed and verified it by running several Spark job and DS jobs. > > BR, > Zhankun > > On Wed, 18 Sep 2019 at 08:05, Naganarasimha Garla < > naganarasimha...@apache.org> wrote: > > > Verified the source and the binary tar and the sha512 checksums > > Installed and verified the basic hadoop operations (ran few MR tasks) > > > > +1. > > > > Thanks, > > + Naga > > > > On Wed, Sep 18, 2019 at 1:32 AM Anil Sadineni > > wrote: > > > > > +1 (non-binding) > > > > > > On Tue, Sep 17, 2019 at 9:55 AM Santosh Marella > > wrote: > > > > > > > +1 (non-binding) > > > > > > > > On Wed, Sep 11, 2019 at 12:26 AM Rohith Sharma K S < > > > > rohithsharm...@apache.org> wrote: > > > > > > > > > Hi folks, > > > > > > > > > > I have put together a release candidate (RC0) for Apache Hadoop > > 3.2.1. > > > > > > > > > > The RC is available at: > > > > > http://home.apache.org/~rohithsharmaks/hadoop-3.2.1-RC0/ > > > > > > > > > > The RC tag in git is release-3.2.1-RC0: > > > > > https://github.com/apache/hadoop/tree/release-3.2.1-RC0 > > > > > > > > > > > > > > > The maven artifacts are staged at > > > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1226/ > > > > > > > > > > You can find my public key at: > > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > > > This vote will run for 7 days(5 weekdays), ending on 18th Sept at > > 11:59 > > > > pm > > > > > PST. > > > > > > > > > > I have done testing with a pseudo cluster and distributed shell job. > > My > > > > +1 > > > > > to start. > > > > > > > > > > Thanks & Regards > > > > > Rohith Sharma K S > > > > > > > > > > > > > > > > > > -- > > > Thanks & Regards, > > > Anil Sadineni > > > Solutions Architect, Optlin Inc > > > Ph: 571-438-1974 | www.optlin.com > > > > >
Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree
one thing to consider here as you are giving up your ability to make changes in hadoop-* modules, including hadoop-common, and their dependencies, in sync with your own code. That goes for filesystem contract tests. are you happy with that? On Tue, Sep 17, 2019 at 10:48 AM Elek, Marton wrote: > > > TLDR; I propose to move Ozone related code out from Hadoop trunk and > store it in a separated *Hadoop* git repository apache/hadoop-ozone.git > > > > > When Ozone was adopted as a new Hadoop subproject it was proposed[1] to > be part of the source tree but with separated release cadence, mainly > because it had the hadoop-trunk/SNAPSHOT as compile time dependency. > > During the last Ozone releases this dependency is removed to provide > more stable releases. Instead of using the latest trunk/SNAPSHOT build > from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now). > > As we have no more strict dependency between Hadoop trunk SNAPSHOT and > Ozone trunk I propose to separate the two code base from each other with > creating a new Hadoop git repository (apache/hadoop-ozone.git): > > With moving Ozone to a separated git repository: > > * It would be easier to contribute and understand the build (as of now > we always need `-f pom.ozone.xml` as a Maven parameter) > * It would be possible to adjust build process without breaking > Hadoop/Ozone builds. > * It would be possible to use different Readme/.asf.yaml/github > template for the Hadoop Ozone and core Hadoop. (For example the current > github template [2] has a link to the contribution guideline [3]. Ozone > has an extended version [4] from this guideline with additional > information.) > * Testing would be more safe as it won't be possible to change core > Hadoop and Hadoop Ozone in the same patch. > * It would be easier to cut branches for Hadoop releases (based on the > original consensus, Ozone should be removed from all the release > branches after creating relase branches from trunk) > > > What do you think? > > Thanks, > Marton > > [1]: > > https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E > [2]: > > https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md > [3]: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute > [4]: > > https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
Re: [VOTE] Release Hadoop-3.1.3-RC0
Hi Masatake, My bad. I read it wrong. Yeah. You're right. The HDFS-14759 can be backported to both Branch-3.2 and 3.1. Let's see if there's any other blocker/critical issue come up, otherwise, personally I don't prefer to run another RC1 and vote for this minor change. BR, Zhankun On Wed, 18 Sep 2019 at 15:02, Masatake Iwasaki wrote: > Hi Zhankun, > > > Can you please help to provide a branch-3.1 patch for HDFS-14759? Or > we can > > move it to the next release of branch-3.1 since the noisy info is not a > > blocker issue to me. Does that make sense? > > I tried to cherry-picking the HDFS-14759 on my side. > Since there was no conflict, I pushed the branch-3.1 to apache repo. > Could you pull and check it? > > Masatake > > On 9/18/19 15:49, Zhankun Tang wrote: > > Hi Masatake, > > > > Thanks for helping to verify! > > I checked that branch-3.2 has the HDFS-14759 committed already. > > Release-3.2.1-RC0 should have no such issue. > > > > For branch-3.1, cherry-pick the same commit has conflicts. I'm confirming > > if we can fix it or there's a feasible plan to backport the whole > > Hadoop-15226 (which targeting branch-3.2) to branch-3.1. > > > > Can you please help to provide a branch-3.1 patch for HDFS-14759? Or we > can > > move it to the next release of branch-3.1 since the noisy info is not a > > blocker issue to me. Does that make sense? > > > > > > BR, > > Zhankun > > > > On Wed, 18 Sep 2019 at 14:23, Masatake Iwasaki < > iwasak...@oss.nttdata.co.jp> > > wrote: > > > >> Thanks for putting this up, Zhankun Tang. > >> > >> While I was testing the RC0 wich CLI, > >> noisy INFO message was emitted on every data transfer operation.:: > >> > >> 2019-09-17 16:00:42,942 INFO sasl.SaslDataTransferClient: SASL > >> encryption trust check: localHostTrusted = false, remoteHostTrusted = > false > >> > >> The issue was fixed by HDFS-14759. > >> I think the fix should be backported if we cut RC1. > >> > >> Since the fix version of HDFS-14759 is 3.3.0, CR0 of 3.2.1 could have > >> same issue. > >> > >> Regards, > >> Masatake Iwasaki > >> > >> On 9/12/19 17:04, Zhankun Tang wrote: > >>> Hi folks, > >>> > >>> Thanks to everyone's help on this release. Special thanks to Rohith, > >>> Wei-Chiu, Akira, Sunil, Wangda! > >>> > >>> I have created a release candidate (RC0) for Apache Hadoop 3.1.3. > >>> > >>> The RC release artifacts are available at: > >>> http://home.apache.org/~ztang/hadoop-3.1.3-RC0/ > >>> > >>> The maven artifacts are staged at: > >>> > https://repository.apache.org/content/repositories/orgapachehadoop-1228/ > >>> > >>> The RC tag in git is here: > >>> https://github.com/apache/hadoop/tree/release-3.1.3-RC0 > >>> > >>> And my public key is at: > >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > >>> > >>> *This vote will run for 7 days, ending on Sept.19th at 11:59 pm PST.* > >>> > >>> For the testing, I have run several Spark and distributed shell jobs in > >> my > >>> pseudo cluster. > >>> > >>> My +1 (non-binding) to start. > >>> > >>> BR, > >>> Zhankun > >>> > >>> On Wed, 4 Sep 2019 at 15:56, zhankun tang > wrote: > >>> > Hi all, > > Thanks for everyone helping in resolving all the blockers targeting > >> Hadoop > 3.1.3[1]. We've cleaned all the blockers and moved out non-blockers > >> issues > to 3.1.4. > > I'll cut the branch today and call a release vote soon. Thanks! > > > [1]. https://s.apache.org/5hj5i > > BR, > Zhankun > > > On Wed, 21 Aug 2019 at 12:38, Zhankun Tang wrote: > > > Hi folks, > > > > We have Apache Hadoop 3.1.2 released on Feb 2019. > > > > It's been more than 6 months passed and there're > > > > 246 fixes[1]. 2 blocker and 4 critical Issues [2] > > > > (As Wei-Chiu Chuang mentioned, HDFS-13596 will be another blocker) > > > > > > I propose my plan to do a maintenance release of 3.1.3 in the next > few > > (one or two) weeks. > > > > Hadoop 3.1.3 release plan: > > > > Code Freezing Date: *25th August 2019 PDT* > > > > Release Date: *31th August 2019 PDT* > > > > > > Please feel free to share your insights on this. Thanks! > > > > > > [1] https://s.apache.org/zw8l5 > > > > [2] https://s.apache.org/fjol5 > > > > > > BR, > > > > Zhankun > > > >> > >
Re: [VOTE] Release Hadoop-3.1.3-RC0
Hi Zhankun, > Can you please help to provide a branch-3.1 patch for HDFS-14759? Or we can > move it to the next release of branch-3.1 since the noisy info is not a > blocker issue to me. Does that make sense? I tried to cherry-picking the HDFS-14759 on my side. Since there was no conflict, I pushed the branch-3.1 to apache repo. Could you pull and check it? Masatake On 9/18/19 15:49, Zhankun Tang wrote: Hi Masatake, Thanks for helping to verify! I checked that branch-3.2 has the HDFS-14759 committed already. Release-3.2.1-RC0 should have no such issue. For branch-3.1, cherry-pick the same commit has conflicts. I'm confirming if we can fix it or there's a feasible plan to backport the whole Hadoop-15226 (which targeting branch-3.2) to branch-3.1. Can you please help to provide a branch-3.1 patch for HDFS-14759? Or we can move it to the next release of branch-3.1 since the noisy info is not a blocker issue to me. Does that make sense? BR, Zhankun On Wed, 18 Sep 2019 at 14:23, Masatake Iwasaki wrote: Thanks for putting this up, Zhankun Tang. While I was testing the RC0 wich CLI, noisy INFO message was emitted on every data transfer operation.:: 2019-09-17 16:00:42,942 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false The issue was fixed by HDFS-14759. I think the fix should be backported if we cut RC1. Since the fix version of HDFS-14759 is 3.3.0, CR0 of 3.2.1 could have same issue. Regards, Masatake Iwasaki On 9/12/19 17:04, Zhankun Tang wrote: Hi folks, Thanks to everyone's help on this release. Special thanks to Rohith, Wei-Chiu, Akira, Sunil, Wangda! I have created a release candidate (RC0) for Apache Hadoop 3.1.3. The RC release artifacts are available at: http://home.apache.org/~ztang/hadoop-3.1.3-RC0/ The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1228/ The RC tag in git is here: https://github.com/apache/hadoop/tree/release-3.1.3-RC0 And my public key is at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS *This vote will run for 7 days, ending on Sept.19th at 11:59 pm PST.* For the testing, I have run several Spark and distributed shell jobs in my pseudo cluster. My +1 (non-binding) to start. BR, Zhankun On Wed, 4 Sep 2019 at 15:56, zhankun tang wrote: Hi all, Thanks for everyone helping in resolving all the blockers targeting Hadoop 3.1.3[1]. We've cleaned all the blockers and moved out non-blockers issues to 3.1.4. I'll cut the branch today and call a release vote soon. Thanks! [1]. https://s.apache.org/5hj5i BR, Zhankun On Wed, 21 Aug 2019 at 12:38, Zhankun Tang wrote: Hi folks, We have Apache Hadoop 3.1.2 released on Feb 2019. It's been more than 6 months passed and there're 246 fixes[1]. 2 blocker and 4 critical Issues [2] (As Wei-Chiu Chuang mentioned, HDFS-13596 will be another blocker) I propose my plan to do a maintenance release of 3.1.3 in the next few (one or two) weeks. Hadoop 3.1.3 release plan: Code Freezing Date: *25th August 2019 PDT* Release Date: *31th August 2019 PDT* Please feel free to share your insights on this. Thanks! [1] https://s.apache.org/zw8l5 [2] https://s.apache.org/fjol5 BR, Zhankun - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
