Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

2019-01-15 Thread Brahma Reddy Battula
Sunil thanks for driving this.

+1 ,(binding)

--Built from Source.
--Installed pseudo cluster.
--Ran basic shell commands
--Ran sample jobs pi,wordcount.



On Tue, Jan 8, 2019 at 5:12 PM Sunil G  wrote:

> Hi folks,
>
>
> Thanks to all of you who helped in this release [1] and for helping to vote
> for RC0. I have created second release candidate (RC1) for Apache Hadoop
> 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
>
>
> RC tag in git is release-3.2.0-RC1.
>
>
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
>
>
> This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>
> 7. Improvements in Router-based HDFS federation
>
>
>
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>
> I have done few testing with my pseudo cluster. My +1 to start.
>
>
>
> Regards,
>
> Sunil
>
>
>
> [1]
>
>
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC
>


-- 
--Brahma Reddy Battula


Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

2019-01-15 Thread Eric Payne
+1 (binding)

- RM refresh updates values as expected
- Streaming jobs complete successfully
- Moving apps between queues succeeds
- Inter-queue preemption works as expected
- Successfully ran selected yarn unit tests.

===
Eric Payne
===


On Tuesday, January 8, 2019, 5:42:46 AM CST, Sunil G  wrote: 





Hi folks,


Thanks to all of you who helped in this release [1] and for helping to vote
for RC0. I have created second release candidate (RC1) for Apache Hadoop
3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/


RC tag in git is release-3.2.0-RC1.



The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1178/


This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST.



3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org



Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

2019-01-15 Thread Wangda Tan
Manually checked the new repo, looks good, thanks Sunil!  +1

Best,
Wangda

On Tue, Jan 15, 2019 at 2:04 PM Sunil G  wrote:

> Thanks folks for voting.
>
> For the point mentioned from Zoltan, I re-ran the same source and deployed
> to Nexus to avail those missing source.
> https://repository.apache.org/content/repositories/orgapachehadoop-1186/
>
> Please help to cross check the same.
>
> Thanks & Regards
> Sunil
>
> On Tue, Jan 15, 2019 at 10:05 AM Wangda Tan  wrote:
>
>> +1 (Binding).
>>
>> Deployed a local cluster from binary, and ran some sample sanity jobs.
>>
>> Thanks Sunil for driving the release.
>>
>> Best,
>> Wangda
>>
>>
>> On Mon, Jan 14, 2019 at 11:26 AM Virajith Jalaparti 
>> wrote:
>>
>>> Thanks Sunil and others who have worked on the making this release
>>> happen!
>>>
>>> +1 (non-binding)
>>>
>>> - Built from source
>>> - Deployed a pseudo-distributed one node cluster
>>> - Ran basic wordcount, sort, pi jobs
>>> - Basic HDFS/WebHDFS commands
>>> - Ran all the ABFS driver tests against an ADLS Gen 2 account in EAST US
>>>
>>> Non-blockers (AFAICT): The following tests in ABFS (HADOOP-15407) fail:
>>> - For ACLs ({{ITestAzureBlobFilesystemAcl}}) -- However, I believe these
>>> have been fixed in trunk.
>>> - {{
>>> ITestAzureBlobFileSystemE2EScale#testWriteHeavyBytesToFileAcrossThreads}}
>>> fails with an OutOfMemoryError exception. I see the same failure on
>>> trunk as well.
>>>
>>>
>>> On Mon, Jan 14, 2019 at 6:21 AM Elek, Marton  wrote:
>>>
 Thanks Sunil to manage this release.

 +1 (non-binding)

 1. built from the source (with clean local maven repo)
 2. verified signatures + checksum
 3. deployed 3 node cluster to Google Kubernetes Engine with generated
 k8s resources [1]
 4. Executed basic HDFS commands
 5. Executed basic yarn example jobs

 Marton

 [1]: FTR: resources:
 https://github.com/flokkr/k8s/tree/master/examples/hadoop , generator:
 https://github.com/elek/flekszible


 On 1/8/19 12:42 PM, Sunil G wrote:
 > Hi folks,
 >
 >
 > Thanks to all of you who helped in this release [1] and for helping
 to vote
 > for RC0. I have created second release candidate (RC1) for Apache
 Hadoop
 > 3.2.0.
 >
 >
 > Artifacts for this RC are available here:
 >
 > http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
 >
 >
 > RC tag in git is release-3.2.0-RC1.
 >
 >
 >
 > The maven artifacts are available via repository.apache.org at
 >
 https://repository.apache.org/content/repositories/orgapachehadoop-1178/
 >
 >
 > This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59
 pm PST.
 >
 >
 >
 > 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
 > additions
 >
 > are the highlights of this release.
 >
 > 1. Node Attributes Support in YARN
 >
 > 2. Hadoop Submarine project for running Deep Learning workloads on
 YARN
 >
 > 3. Support service upgrade via YARN Service API and CLI
 >
 > 4. HDFS Storage Policy Satisfier
 >
 > 5. Support Windows Azure Storage - Blob file system in Hadoop
 >
 > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
 >
 > 7. Improvements in Router-based HDFS federation
 >
 >
 >
 > Thanks to Wangda, Vinod, Marton for helping me in preparing the
 release.
 >
 > I have done few testing with my pseudo cluster. My +1 to start.
 >
 >
 >
 > Regards,
 >
 > Sunil
 >
 >
 >
 > [1]
 >
 >
 https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
 >
 > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
 (3.2.0)
 > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status =
 Resolved
 > ORDER BY fixVersion ASC
 >

 -
 To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
 For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org




Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

2019-01-15 Thread Sunil G
Thanks folks for voting.

For the point mentioned from Zoltan, I re-ran the same source and deployed
to Nexus to avail those missing source.
https://repository.apache.org/content/repositories/orgapachehadoop-1186/

Please help to cross check the same.

Thanks & Regards
Sunil

On Tue, Jan 15, 2019 at 10:05 AM Wangda Tan  wrote:

> +1 (Binding).
>
> Deployed a local cluster from binary, and ran some sample sanity jobs.
>
> Thanks Sunil for driving the release.
>
> Best,
> Wangda
>
>
> On Mon, Jan 14, 2019 at 11:26 AM Virajith Jalaparti 
> wrote:
>
>> Thanks Sunil and others who have worked on the making this release happen!
>>
>> +1 (non-binding)
>>
>> - Built from source
>> - Deployed a pseudo-distributed one node cluster
>> - Ran basic wordcount, sort, pi jobs
>> - Basic HDFS/WebHDFS commands
>> - Ran all the ABFS driver tests against an ADLS Gen 2 account in EAST US
>>
>> Non-blockers (AFAICT): The following tests in ABFS (HADOOP-15407) fail:
>> - For ACLs ({{ITestAzureBlobFilesystemAcl}}) -- However, I believe these
>> have been fixed in trunk.
>> - {{
>> ITestAzureBlobFileSystemE2EScale#testWriteHeavyBytesToFileAcrossThreads}}
>> fails with an OutOfMemoryError exception. I see the same failure on
>> trunk as well.
>>
>>
>> On Mon, Jan 14, 2019 at 6:21 AM Elek, Marton  wrote:
>>
>>> Thanks Sunil to manage this release.
>>>
>>> +1 (non-binding)
>>>
>>> 1. built from the source (with clean local maven repo)
>>> 2. verified signatures + checksum
>>> 3. deployed 3 node cluster to Google Kubernetes Engine with generated
>>> k8s resources [1]
>>> 4. Executed basic HDFS commands
>>> 5. Executed basic yarn example jobs
>>>
>>> Marton
>>>
>>> [1]: FTR: resources:
>>> https://github.com/flokkr/k8s/tree/master/examples/hadoop , generator:
>>> https://github.com/elek/flekszible
>>>
>>>
>>> On 1/8/19 12:42 PM, Sunil G wrote:
>>> > Hi folks,
>>> >
>>> >
>>> > Thanks to all of you who helped in this release [1] and for helping to
>>> vote
>>> > for RC0. I have created second release candidate (RC1) for Apache
>>> Hadoop
>>> > 3.2.0.
>>> >
>>> >
>>> > Artifacts for this RC are available here:
>>> >
>>> > http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
>>> >
>>> >
>>> > RC tag in git is release-3.2.0-RC1.
>>> >
>>> >
>>> >
>>> > The maven artifacts are available via repository.apache.org at
>>> >
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
>>> >
>>> >
>>> > This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm
>>> PST.
>>> >
>>> >
>>> >
>>> > 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
>>> > additions
>>> >
>>> > are the highlights of this release.
>>> >
>>> > 1. Node Attributes Support in YARN
>>> >
>>> > 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>>> >
>>> > 3. Support service upgrade via YARN Service API and CLI
>>> >
>>> > 4. HDFS Storage Policy Satisfier
>>> >
>>> > 5. Support Windows Azure Storage - Blob file system in Hadoop
>>> >
>>> > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>>> >
>>> > 7. Improvements in Router-based HDFS federation
>>> >
>>> >
>>> >
>>> > Thanks to Wangda, Vinod, Marton for helping me in preparing the
>>> release.
>>> >
>>> > I have done few testing with my pseudo cluster. My +1 to start.
>>> >
>>> >
>>> >
>>> > Regards,
>>> >
>>> > Sunil
>>> >
>>> >
>>> >
>>> > [1]
>>> >
>>> >
>>> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>>> >
>>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>>> (3.2.0)
>>> > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
>>> > ORDER BY fixVersion ASC
>>> >
>>>
>>> -
>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>>
>>>


Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

2019-01-15 Thread Iñigo Goiri
+1 (binding)

- Deployed a cluster with 3 NNs, 3 RMs and 1 DN/NM on Azure
- Tested the Active probe for the Load Balancer in front of the NNs and the
RMs
- Checked the NN, RBF, and RM Web UIs
- Executed a TeraGen, TeraSort  and TeraValidate
- Executed a YARN service with a TensorFlow app on Docker
- Scaled up the cluster to 5 DNs/NMs and executed the tests again
- Checked the HDFS output folders through the Web UI

On Tue, Jan 15, 2019 at 9:05 AM Wangda Tan  wrote:

> +1 (Binding).
>
> Deployed a local cluster from binary, and ran some sample sanity jobs.
>
> Thanks Sunil for driving the release.
>
> Best,
> Wangda
>
>
> On Mon, Jan 14, 2019 at 11:26 AM Virajith Jalaparti 
> wrote:
>
> > Thanks Sunil and others who have worked on the making this release
> happen!
> >
> > +1 (non-binding)
> >
> > - Built from source
> > - Deployed a pseudo-distributed one node cluster
> > - Ran basic wordcount, sort, pi jobs
> > - Basic HDFS/WebHDFS commands
> > - Ran all the ABFS driver tests against an ADLS Gen 2 account in EAST US
> >
> > Non-blockers (AFAICT): The following tests in ABFS (HADOOP-15407) fail:
> > - For ACLs ({{ITestAzureBlobFilesystemAcl}}) -- However, I believe these
> > have been fixed in trunk.
> > - {{
> > ITestAzureBlobFileSystemE2EScale#testWriteHeavyBytesToFileAcrossThreads}}
> > fails with an OutOfMemoryError exception. I see the same failure on trunk
> > as well.
> >
> >
> > On Mon, Jan 14, 2019 at 6:21 AM Elek, Marton  wrote:
> >
> >> Thanks Sunil to manage this release.
> >>
> >> +1 (non-binding)
> >>
> >> 1. built from the source (with clean local maven repo)
> >> 2. verified signatures + checksum
> >> 3. deployed 3 node cluster to Google Kubernetes Engine with generated
> >> k8s resources [1]
> >> 4. Executed basic HDFS commands
> >> 5. Executed basic yarn example jobs
> >>
> >> Marton
> >>
> >> [1]: FTR: resources:
> >> https://github.com/flokkr/k8s/tree/master/examples/hadoop , generator:
> >> https://github.com/elek/flekszible
> >>
> >>
> >> On 1/8/19 12:42 PM, Sunil G wrote:
> >> > Hi folks,
> >> >
> >> >
> >> > Thanks to all of you who helped in this release [1] and for helping to
> >> vote
> >> > for RC0. I have created second release candidate (RC1) for Apache
> Hadoop
> >> > 3.2.0.
> >> >
> >> >
> >> > Artifacts for this RC are available here:
> >> >
> >> > http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
> >> >
> >> >
> >> > RC tag in git is release-3.2.0-RC1.
> >> >
> >> >
> >> >
> >> > The maven artifacts are available via repository.apache.org at
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
> >> >
> >> >
> >> > This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm
> >> PST.
> >> >
> >> >
> >> >
> >> > 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
> >> > additions
> >> >
> >> > are the highlights of this release.
> >> >
> >> > 1. Node Attributes Support in YARN
> >> >
> >> > 2. Hadoop Submarine project for running Deep Learning workloads on
> YARN
> >> >
> >> > 3. Support service upgrade via YARN Service API and CLI
> >> >
> >> > 4. HDFS Storage Policy Satisfier
> >> >
> >> > 5. Support Windows Azure Storage - Blob file system in Hadoop
> >> >
> >> > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> >> >
> >> > 7. Improvements in Router-based HDFS federation
> >> >
> >> >
> >> >
> >> > Thanks to Wangda, Vinod, Marton for helping me in preparing the
> release.
> >> >
> >> > I have done few testing with my pseudo cluster. My +1 to start.
> >> >
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Sunil
> >> >
> >> >
> >> >
> >> > [1]
> >> >
> >> >
> >>
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> >> >
> >> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.2.0)
> >> > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status =
> Resolved
> >> > ORDER BY fixVersion ASC
> >> >
> >>
> >> -
> >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >>
> >>
>


Re: [Urgent] Question about Nexus repo and Hadoop release

2019-01-15 Thread Wangda Tan
It seems the problem still exists for me:

Now the error message only contains:

failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-client-check-invariants/3.1.2/hadoop-client-check-invariants-3.1.2.pom',
check the logs.
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-resourceestimator/3.1.2/hadoop-resourceestimator-3.1.2-javadoc.jar',
check the logs.

If anybody has access the Nexus node, could you please help to check what
is the failure message?

Thanks,
Wangda


On Tue, Jan 15, 2019 at 9:56 AM Brian Fox  wrote:

> Good to know. The pool has occasionally had sync issues, but we're talking
> 3 times in the last 8-9 years.
>
> On Tue, Jan 15, 2019 at 10:39 AM Elek, Marton  wrote:
>
>> My key was pushed to the server with pgp about 1 year ago, and it worked
>> well with the last Ratis release. So it should be synced between the key
>> servers.
>>
>> But it seems that the INFRA solved the problem with shuffling the key
>> server order (or it was an intermittent issue): see INFRA-17649
>>
>> Seems to be working now...
>>
>> Marton
>>
>>
>> On 1/15/19 5:19 AM, Wangda Tan wrote:
>> > HI Brain,
>> > Thanks for responding, could u share how to push to keys to Apache pgp
>> pool?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jan 14, 2019 at 10:44 AM Brian Fox  wrote:
>> >
>> >> Did you push your key up to the pgp pool? That's what Nexus is
>> validating
>> >> against. It might take time to propagate if you just pushed it.
>> >>
>> >> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton  wrote:
>> >>
>> >>> Seems to be an INFRA issue for me:
>> >>>
>> >>> 1. I downloaded a sample jar file [1] + the signature from the
>> >>> repository and it was ok, locally I verified it.
>> >>>
>> >>> 2. I tested it with an other Apache project (Ratis) and my key. I got
>> >>> the same problem even if it worked at last year during the 0.3.0
>> >>> release. (I used exactly the same command)
>> >>>
>> >>> I opened an infra ticket to check the logs of the Nexus as it was
>> >>> suggested in the error message:
>> >>>
>> >>> https://issues.apache.org/jira/browse/INFRA-17649
>> >>>
>> >>> Marton
>> >>>
>> >>>
>> >>> [1]:
>> >>>
>> >>>
>> https://repository.apache.org/service/local/repositories/orgapachehadoop-1183/content/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-javadoc.jar
>> >>>
>> >>>
>> >>> On 1/13/19 6:27 AM, Wangda Tan wrote:
>>  Uploaded sample file and signature.
>> 
>> 
>> 
>>  On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan >  > wrote:
>> 
>>  Actually, among the hundreds of failed messages, the "No public
>> key"
>>  issues still occurred several times:
>> 
>>  failureMessage  No public key: Key with id:
>> (b3fa653d57300d45)
>>  was not able to be located on http://gpg-keyserver.de/.
>> Upload
>>  your public key and try the operation again.
>>  failureMessage  No public key: Key with id:
>> (b3fa653d57300d45)
>>  was not able to be located on
>>  http://pool.sks-keyservers.net:11371. Upload your public key
>> >>> and
>>  try the operation again.
>>  failureMessage  No public key: Key with id:
>> (b3fa653d57300d45)
>>  was not able to be located on http://pgp.mit.edu:11371.
>> Upload
>>  your public key and try the operation again.
>> 
>>  Once the close operation returned, I will upload sample files
>> which
>>  may help troubleshoot the issue.
>> 
>>  Thanks,
>> 
>>  On Sat, Jan 12, 2019 at 9:04 PM Wangda Tan >  > wrote:
>> 
>>  Thanks David for the quick response!
>> 
>>  I just retried, now the "No public key" issue is gone.
>> However,
>>  the issue:
>> 
>>  failureMessage  Failed to validate the pgp signature of
>> 
>> >>>
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar',
>>  check the logs.
>>  failureMessage  Failed to validate the pgp signature of
>> 
>> >>>
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-test-sources.jar',
>>  check the logs.
>>  failureMessage  Failed to validate the pgp signature of
>> 
>> >>>
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2.pom',
>>  check the logs.
>> 
>> 
>>  Still exists and repeated hundreds of times. Do you know how
>> to
>>  access the logs mentioned by above log?
>> 
>>  Best,
>>  Wangda
>> 
>>  On Sat, Jan 12, 2019 at 8:37 PM David Nalley >  > wrote:
>> 

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

2019-01-15 Thread Wangda Tan
+1 (Binding).

Deployed a local cluster from binary, and ran some sample sanity jobs.

Thanks Sunil for driving the release.

Best,
Wangda


On Mon, Jan 14, 2019 at 11:26 AM Virajith Jalaparti 
wrote:

> Thanks Sunil and others who have worked on the making this release happen!
>
> +1 (non-binding)
>
> - Built from source
> - Deployed a pseudo-distributed one node cluster
> - Ran basic wordcount, sort, pi jobs
> - Basic HDFS/WebHDFS commands
> - Ran all the ABFS driver tests against an ADLS Gen 2 account in EAST US
>
> Non-blockers (AFAICT): The following tests in ABFS (HADOOP-15407) fail:
> - For ACLs ({{ITestAzureBlobFilesystemAcl}}) -- However, I believe these
> have been fixed in trunk.
> - {{
> ITestAzureBlobFileSystemE2EScale#testWriteHeavyBytesToFileAcrossThreads}}
> fails with an OutOfMemoryError exception. I see the same failure on trunk
> as well.
>
>
> On Mon, Jan 14, 2019 at 6:21 AM Elek, Marton  wrote:
>
>> Thanks Sunil to manage this release.
>>
>> +1 (non-binding)
>>
>> 1. built from the source (with clean local maven repo)
>> 2. verified signatures + checksum
>> 3. deployed 3 node cluster to Google Kubernetes Engine with generated
>> k8s resources [1]
>> 4. Executed basic HDFS commands
>> 5. Executed basic yarn example jobs
>>
>> Marton
>>
>> [1]: FTR: resources:
>> https://github.com/flokkr/k8s/tree/master/examples/hadoop , generator:
>> https://github.com/elek/flekszible
>>
>>
>> On 1/8/19 12:42 PM, Sunil G wrote:
>> > Hi folks,
>> >
>> >
>> > Thanks to all of you who helped in this release [1] and for helping to
>> vote
>> > for RC0. I have created second release candidate (RC1) for Apache Hadoop
>> > 3.2.0.
>> >
>> >
>> > Artifacts for this RC are available here:
>> >
>> > http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
>> >
>> >
>> > RC tag in git is release-3.2.0-RC1.
>> >
>> >
>> >
>> > The maven artifacts are available via repository.apache.org at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
>> >
>> >
>> > This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm
>> PST.
>> >
>> >
>> >
>> > 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
>> > additions
>> >
>> > are the highlights of this release.
>> >
>> > 1. Node Attributes Support in YARN
>> >
>> > 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>> >
>> > 3. Support service upgrade via YARN Service API and CLI
>> >
>> > 4. HDFS Storage Policy Satisfier
>> >
>> > 5. Support Windows Azure Storage - Blob file system in Hadoop
>> >
>> > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>> >
>> > 7. Improvements in Router-based HDFS federation
>> >
>> >
>> >
>> > Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>> >
>> > I have done few testing with my pseudo cluster. My +1 to start.
>> >
>> >
>> >
>> > Regards,
>> >
>> > Sunil
>> >
>> >
>> >
>> > [1]
>> >
>> >
>> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
>> > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
>> > ORDER BY fixVersion ASC
>> >
>>
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>
>>


Re: [Urgent] Question about Nexus repo and Hadoop release

2019-01-15 Thread Brian Fox
Good to know. The pool has occasionally had sync issues, but we're talking
3 times in the last 8-9 years.

On Tue, Jan 15, 2019 at 10:39 AM Elek, Marton  wrote:

> My key was pushed to the server with pgp about 1 year ago, and it worked
> well with the last Ratis release. So it should be synced between the key
> servers.
>
> But it seems that the INFRA solved the problem with shuffling the key
> server order (or it was an intermittent issue): see INFRA-17649
>
> Seems to be working now...
>
> Marton
>
>
> On 1/15/19 5:19 AM, Wangda Tan wrote:
> > HI Brain,
> > Thanks for responding, could u share how to push to keys to Apache pgp
> pool?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jan 14, 2019 at 10:44 AM Brian Fox  wrote:
> >
> >> Did you push your key up to the pgp pool? That's what Nexus is
> validating
> >> against. It might take time to propagate if you just pushed it.
> >>
> >> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton  wrote:
> >>
> >>> Seems to be an INFRA issue for me:
> >>>
> >>> 1. I downloaded a sample jar file [1] + the signature from the
> >>> repository and it was ok, locally I verified it.
> >>>
> >>> 2. I tested it with an other Apache project (Ratis) and my key. I got
> >>> the same problem even if it worked at last year during the 0.3.0
> >>> release. (I used exactly the same command)
> >>>
> >>> I opened an infra ticket to check the logs of the Nexus as it was
> >>> suggested in the error message:
> >>>
> >>> https://issues.apache.org/jira/browse/INFRA-17649
> >>>
> >>> Marton
> >>>
> >>>
> >>> [1]:
> >>>
> >>>
> https://repository.apache.org/service/local/repositories/orgapachehadoop-1183/content/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-javadoc.jar
> >>>
> >>>
> >>> On 1/13/19 6:27 AM, Wangda Tan wrote:
>  Uploaded sample file and signature.
> 
> 
> 
>  On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan   > wrote:
> 
>  Actually, among the hundreds of failed messages, the "No public
> key"
>  issues still occurred several times:
> 
>  failureMessage  No public key: Key with id: (b3fa653d57300d45)
>  was not able to be located on http://gpg-keyserver.de/.
> Upload
>  your public key and try the operation again.
>  failureMessage  No public key: Key with id: (b3fa653d57300d45)
>  was not able to be located on
>  http://pool.sks-keyservers.net:11371. Upload your public key
> >>> and
>  try the operation again.
>  failureMessage  No public key: Key with id: (b3fa653d57300d45)
>  was not able to be located on http://pgp.mit.edu:11371.
> Upload
>  your public key and try the operation again.
> 
>  Once the close operation returned, I will upload sample files
> which
>  may help troubleshoot the issue.
> 
>  Thanks,
> 
>  On Sat, Jan 12, 2019 at 9:04 PM Wangda Tan   > wrote:
> 
>  Thanks David for the quick response!
> 
>  I just retried, now the "No public key" issue is gone.
> However,
>  the issue:
> 
>  failureMessage  Failed to validate the pgp signature of
> 
> >>>
> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar',
>  check the logs.
>  failureMessage  Failed to validate the pgp signature of
> 
> >>>
> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-test-sources.jar',
>  check the logs.
>  failureMessage  Failed to validate the pgp signature of
> 
> >>>
> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2.pom',
>  check the logs.
> 
> 
>  Still exists and repeated hundreds of times. Do you know how
> to
>  access the logs mentioned by above log?
> 
>  Best,
>  Wangda
> 
>  On Sat, Jan 12, 2019 at 8:37 PM David Nalley   > wrote:
> 
>  On Sat, Jan 12, 2019 at 9:09 PM Wangda Tan
>  mailto:wheele...@gmail.com>> wrote:
>  >
>  > Hi Devs,
>  >
>  > I'm currently rolling Hadoop 3.1.2 release candidate,
>  however, I saw an issue when I try to close repo in Nexus.
>  >
>  > Logs of
> >>> https://repository.apache.org/#stagingRepositories
>  (orgapachehadoop-1183) shows hundreds of lines of the
>  following error:
>  >
>  > failureMessage  No public key: Key with id:
>  (b3fa653d57300d45) was not able to be located on
>    

Re: [Urgent] Question about Nexus repo and Hadoop release

2019-01-15 Thread Elek, Marton
My key was pushed to the server with pgp about 1 year ago, and it worked
well with the last Ratis release. So it should be synced between the key
servers.

But it seems that the INFRA solved the problem with shuffling the key
server order (or it was an intermittent issue): see INFRA-17649

Seems to be working now...

Marton


On 1/15/19 5:19 AM, Wangda Tan wrote:
> HI Brain,
> Thanks for responding, could u share how to push to keys to Apache pgp pool?
> 
> Best,
> Wangda
> 
> On Mon, Jan 14, 2019 at 10:44 AM Brian Fox  wrote:
> 
>> Did you push your key up to the pgp pool? That's what Nexus is validating
>> against. It might take time to propagate if you just pushed it.
>>
>> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton  wrote:
>>
>>> Seems to be an INFRA issue for me:
>>>
>>> 1. I downloaded a sample jar file [1] + the signature from the
>>> repository and it was ok, locally I verified it.
>>>
>>> 2. I tested it with an other Apache project (Ratis) and my key. I got
>>> the same problem even if it worked at last year during the 0.3.0
>>> release. (I used exactly the same command)
>>>
>>> I opened an infra ticket to check the logs of the Nexus as it was
>>> suggested in the error message:
>>>
>>> https://issues.apache.org/jira/browse/INFRA-17649
>>>
>>> Marton
>>>
>>>
>>> [1]:
>>>
>>> https://repository.apache.org/service/local/repositories/orgapachehadoop-1183/content/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-javadoc.jar
>>>
>>>
>>> On 1/13/19 6:27 AM, Wangda Tan wrote:
 Uploaded sample file and signature.



 On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan >>> > wrote:

 Actually, among the hundreds of failed messages, the "No public key"
 issues still occurred several times:

 failureMessage  No public key: Key with id: (b3fa653d57300d45)
 was not able to be located on http://gpg-keyserver.de/. Upload
 your public key and try the operation again.
 failureMessage  No public key: Key with id: (b3fa653d57300d45)
 was not able to be located on
 http://pool.sks-keyservers.net:11371. Upload your public key
>>> and
 try the operation again.
 failureMessage  No public key: Key with id: (b3fa653d57300d45)
 was not able to be located on http://pgp.mit.edu:11371. Upload
 your public key and try the operation again.

 Once the close operation returned, I will upload sample files which
 may help troubleshoot the issue.

 Thanks,

 On Sat, Jan 12, 2019 at 9:04 PM Wangda Tan >>> > wrote:

 Thanks David for the quick response!

 I just retried, now the "No public key" issue is gone. However,
 the issue:

 failureMessage  Failed to validate the pgp signature of

>>>  
>>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar',
 check the logs.
 failureMessage  Failed to validate the pgp signature of

>>>  
>>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-test-sources.jar',
 check the logs.
 failureMessage  Failed to validate the pgp signature of

>>>  
>>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2.pom',
 check the logs.


 Still exists and repeated hundreds of times. Do you know how to
 access the logs mentioned by above log?

 Best,
 Wangda

 On Sat, Jan 12, 2019 at 8:37 PM David Nalley >>> > wrote:

 On Sat, Jan 12, 2019 at 9:09 PM Wangda Tan
 mailto:wheele...@gmail.com>> wrote:
 >
 > Hi Devs,
 >
 > I'm currently rolling Hadoop 3.1.2 release candidate,
 however, I saw an issue when I try to close repo in Nexus.
 >
 > Logs of
>>> https://repository.apache.org/#stagingRepositories
 (orgapachehadoop-1183) shows hundreds of lines of the
 following error:
 >
 > failureMessage  No public key: Key with id:
 (b3fa653d57300d45) was not able to be located on
 http://gpg-keyserver.de/. Upload your public key and try
>>> the
 operation again.
 > failureMessage  No public key: Key with id:
 (b3fa653d57300d45) was not able to be located on
 http://pool.sks-keyservers.net:11371. Upload your public
>>> key
 and try the operation again.
 > failureMessage  No public

[ANNOUNCE] Apache Roadshow Chicago, Call for Presentations

2019-01-15 Thread Trevor Grant
Hello Devs!


You're receiving this email because you are subscribed to one or more
Apache developer email lists.

I’m writing to let you know about an exciting event coming to the Chicago
area: The Apache Roadshow Chicago.  It will be held May 13th and 14th at
three bars in the Logan Square neighborhood (Revolution Brewing, The
Native, and the Radler).

There will be six tracks:

   -

   Apache in Adtech:  Tell us how Apache works in your advertising stack
   -

   Apache in Fintech: Tell us how Apache works in your finance/insurance
   business
   -

   Apache in Startups: Tell us how you’re using Apache in your startup
   -

   Diversity in Apache: How do we increase and encourage diversity in
   Apache and tech fields overall?
   -

   Made in Chicago: Apache related things made by people in Chicago that
   don’t fall into other buckets
   -

   Project Shark Tank: Do you want more developers or users for your Apache
   project? Come here and pitch it!


This is an exciting chance to learn about how Apache Projects are in use in
production around Chicago, how business users make the decision to use
Apache projects, to learn about exciting new projects that want help from
developers like you, and how/why to increase diversity in tech and IT.

If you have any use cases of Apache products in Adtech, Fintech, or
Startups; if you represent a minority working in tech and have perspectives
to share, if you live in the Chicagoland area and want to highlight some
work you’ve done on an Apache project, or if you want to get other people
excited to come work on your project, then please submit a CFP before the
deadline on February 15th!

Tickets to the Apache Roadshow Chicago are $100; speakers will get a
complimentary ticket.

We’re looking forward to reading your submissions and seeing you there on
May 13-14!

Sincerely,

Trevor Grant

https://www.apachecon.com/chiroadshow19/cfp.html

https://www.apachecon.com/chiroadshow19/register.html


Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2019-01-15 Thread Apache Jenkins Server
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/

[Jan 14, 2019 1:12:05 PM] (stevel) HADOOP-16043. NPE in 
ITestDynamoDBMetadataStore when
[Jan 14, 2019 5:27:00 PM] (stevel) HADOOP-15843. s3guard bucket-info command to 
not print a stack trace on
[Jan 14, 2019 5:59:27 PM] (stevel) HADOOP-14556. S3A to support Delegation 
Tokens.
[Jan 14, 2019 7:32:56 PM] (stevel) HADOOP-16044. ABFS: Better exception 
handling of DNS errors followup
[Jan 15, 2019 12:30:21 AM] (bharat) HDDS-977. Exclude 
dependency-reduced-pom.xml from ozone rat check.




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.service.TestYarnNativeServices 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/diff-compile-javac-root.txt
  [336K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/diff-patch-pylint.txt
  [60K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/whitespace-eol.txt
  [9.3M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/whitespace-tabs.txt
  [1.1M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-ozone_common.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/branch-findbugs-hadoop-ozone_tools.txt
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/diff-javadoc-javadoc-root.txt
  [752K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [440K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [84K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1017/artifact/out/patch-unit-hadoop-yarn-project_hadoop-ya

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

2019-01-15 Thread Takanobu Asanuma
Thanks for driving the release, Sunil.

+1(non-binding)
 - verified checksums
 - built package from source code
 - started RBF(HDFS) cluster
 - verified Web UI (NN, RM, Router)
 - verified some operations of Router-based Federation
 - verified some operations of Erasure Coding

Regards,
- Takanobu

on 2019/01/08 20:42, "Sunil G" wrote:

Hi folks,


Thanks to all of you who helped in this release [1] and for helping to vote
for RC0. I have created second release candidate (RC1) for Apache Hadoop
3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/


RC tag in git is release-3.2.0-RC1.



The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1178/


This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST.



3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]


https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC