[jira] [Created] (HADOOP-14922) Build of Mapreduce Native Task module fails with unknown opcode "bswap"

[Anup Halarnkar (JIRA)]

Anup Halarnkar created HADOOP-14922:
---

 Summary: Build of Mapreduce Native Task module fails with unknown 
opcode "bswap"
 Key: HADOOP-14922
 URL: https://issues.apache.org/jira/browse/HADOOP-14922
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0-alpha3
 Environment: OS: Ubuntu 14.04
Arch: PPC64LE

Reporter: Anup Halarnkar
 Fix For: 3.0.0-alpha3


[WARNING] /tmp/cckBBdQp.s: Assembler messages:
[WARNING] /tmp/cckBBdQp.s:3127: Error: unrecognized opcode: `bswap'
[WARNING] /tmp/cckBBdQp.s:3152: Error: unrecognized opcode: `bswap'
[WARNING] make[2]: *** 
[CMakeFiles/nativetask.dir/main/native/src/codec/BlockCodec.cc.o] Error 1
[WARNING] make[2]: *** Waiting for unfinished jobs
[WARNING] /tmp/ccqRfBZp.s: Assembler messages:
[WARNING] /tmp/ccqRfBZp.s:2098: Error: unrecognized opcode: `bswap'
[WARNING] /tmp/ccqRfBZp.s:2123: Error: unrecognized opcode: `bswap'
[WARNING] make[2]: *** 
[CMakeFiles/nativetask.dir/main/native/src/codec/Lz4Codec.cc.o] Error 1
[WARNING] /tmp/cc50B5Mp.s: Assembler messages:
[WARNING] /tmp/cc50B5Mp.s:3112: Error: unrecognized opcode: `bswap'
[WARNING] /tmp/cc50B5Mp.s:3137: Error: unrecognized opcode: `bswap'
[WARNING] make[2]: *** 
[CMakeFiles/nativetask_static.dir/main/native/src/codec/BlockCodec.cc.o] Error 1
[WARNING] make[2]: *** Waiting for unfinished jobs
[WARNING] /tmp/ccobJqOY.s: Assembler messages:
[WARNING] /tmp/ccobJqOY.s:2098: Error: unrecognized opcode: `bswap'
[WARNING] /tmp/ccobJqOY.s:2123: Error: unrecognized opcode: `bswap'
[WARNING] make[2]: *** 
[CMakeFiles/nativetask_static.dir/main/native/src/codec/Lz4Codec.cc.o] Error 1
[WARNING] /tmp/ccdaQ1CY.s: Assembler messages:
[WARNING] /tmp/ccdaQ1CY.s:2235: Error: unrecognized opcode: `bswap'
[WARNING] /tmp/ccdaQ1CY.s:2249: Error: unrecognized opcode: `bswap'
[WARNING] /tmp/ccRwHt5X.s: Assembler messages:
[WARNING] /tmp/ccRwHt5X.s:2235: Error: unrecognized opcode: `bswap'
[WARNING] /tmp/ccRwHt5X.s:2249: Error: unrecognized opcode: `bswap'
[WARNING] make[2]: *** 
[CMakeFiles/nativetask.dir/main/native/src/codec/SnappyCodec.cc.o] Error 1
[WARNING] make[1]: *** [CMakeFiles/nativetask.dir/all] Error 2
[WARNING] make[1]: *** Waiting for unfinished jobs
[WARNING] make[2]: *** 
[CMakeFiles/nativetask_static.dir/main/native/src/codec/SnappyCodec.cc.o] Error 
1
[WARNING] make[1]: *** [CMakeFiles/nativetask_static.dir/all] Error 2
[WARNING] make: *** [all] Error 2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran reopened HADOOP-14220:
-

> Enhance S3GuardTool with bucket-info and set-capacity commands, tests
> -
>
> Key: HADOOP-14220
> URL: https://issues.apache.org/jira/browse/HADOOP-14220
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, 
> HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch, 
> HADOOP-14220-013.patch, HADOOP-14220-013.patch, HADOOP-14220-014.patch, 
> HADOOP-14220-015.patch, HADOOP-14220-016.patch, 
> HADOOP-14220-branch-2-016.patch, HADOOP-14220-HADOOP-13345-001.patch, 
> HADOOP-14220-HADOOP-13345-002.patch, HADOOP-14220-HADOOP-13345-003.patch, 
> HADOOP-14220-HADOOP-13345-004.patch, HADOOP-14220-HADOOP-13345-005.patch
>
>
> Add a diagnostics command to s3guard which does whatever we need to diagnose 
> problems for a specific (named) s3a url. This is something which can be 
> attached to bug reports as well as used by developers.
> * Properties to log (with provenance attribute, which can track bucket 
> overrides: s3guard metastore setup, autocreate, capacity, 
> * table present/absent
> * # of keys in DDB table for that bucket?
> * any other stats?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[Marton, Elek]

+1 (non-binding)

* Built from the source
* Installed dockerized YARN/HADOOP cluster to a 20 node cluster 
(scheduled with nomad, configured from consul, docker host networking)
* Started example yarn jobs (teragen/terasort) and hdfs dfs commands + 
checking UIs


I noticed only two very minor issues (changelog of HADOOP-9902 didn't 
mention that I need a writable 'logs' dir, even with custom 
log4j.properties; and there was a space typo in yarn error message 
https://issues.apache.org/jira/browse/YARN-7279)


Marton

On 09/29/2017 02:04 AM, Andrew Wang wrote:

Hi all,

Let me start, as always, by thanking the many, many contributors who helped
with this release! I've prepared an RC0 for 3.0.0-beta1:

http://home.apache.org/~wang/3.0.0-beta1-RC0/

This vote will run five days, ending on Nov 3rd at 5PM Pacific.

beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
improvements, and feature enhancements. Notable additions include the
addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
shaded client, and HDFS erasure coding pluggable policy support.

I've done the traditional testing of running a Pi job on a pseudo cluster.
My +1 to start.

We're working internally on getting this run through our integration test
rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.

Best,
Andrew



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/

[Oct 2, 2017 5:58:19 PM] (cliang) HADOOP-14915. method name is incorrect in 
ConfServlet. Contributed by
[Oct 2, 2017 6:26:53 PM] (cliang) HDFS-12552. Use slf4j instead of log4j in 
FSNamesystem. Contributed by
[Oct 2, 2017 8:03:32 PM] (Carlo Curino) YARN-3661. Basic Federation UI. 
(Contributed by Inigo Goiri via curino)
[Oct 2, 2017 8:11:47 PM] (aengineer) HADOOP-14752. 
TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky.
[Oct 2, 2017 8:18:00 PM] (cliang) HDFS-12271. Incorrect statement in Downgrade 
section of HDFS Rolling
[Oct 2, 2017 10:30:22 PM] (wangda) YARN-7259. Add size-based rolling policy to
[Oct 2, 2017 10:35:49 PM] (lei) HDFS-12569. Unset EC policy logs empty payload 
in edit log. (Lei (Eddy)
[Oct 2, 2017 10:40:00 PM] (aengineer) HDFS-12576. JournalNodes are getting 
started, even though
[Oct 3, 2017 1:14:44 AM] (subru) YARN-2037. Add work preserving restart support 
for Unmanaged AMs.
[Oct 3, 2017 2:04:49 AM] (sidharta) YARN-7226. Whitelisted variables do not 
support delayed variable
[Oct 3, 2017 3:14:54 AM] (aajisaka) MAPREDUCE-6971. Moving logging APIs over to 
slf4j in




-1 overall


The following subsystems voted -1:
unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.net.TestDNS 
   hadoop.crypto.key.kms.server.TestKMS 
   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.sls.TestReservationSystemInvariants 
   hadoop.yarn.sls.TestSLSRunner 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/whitespace-tabs.txt
  [1.2M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [152K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/patch-unit-hadoop-common-project_hadoop-kms.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [296K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [20K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[Rushabh Shah]

Thank you Andrew for all your efforts.
+1 non-binding.
Built from source.
Started 1 node pseudo cluster.
Ran sleep and pi job.
Except for YARN-6999  I
didn't face any issues.


On Thu, Sep 28, 2017 at 7:04 PM, Andrew Wang 
wrote:

> Hi all,
>
> Let me start, as always, by thanking the many, many contributors who helped
> with this release! I've prepared an RC0 for 3.0.0-beta1:
>
> http://home.apache.org/~wang/3.0.0-beta1-RC0/
>
> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
>
> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
> improvements, and feature enhancements. Notable additions include the
> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
> shaded client, and HDFS erasure coding pluggable policy support.
>
> I've done the traditional testing of running a Pi job on a pseudo cluster.
> My +1 to start.
>
> We're working internally on getting this run through our integration test
> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.
>
> Best,
> Andrew
>

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[Brahma Reddy Battula]

Thanks Andrew.

+1 (non binding)

--Built from source
--installed 3 node HA cluster
--Verified shell commands and UI
--Ran wordcount/pic jobs




On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang 
wrote:

> Hi all,
>
> Let me start, as always, by thanking the many, many contributors who helped
> with this release! I've prepared an RC0 for 3.0.0-beta1:
>
> http://home.apache.org/~wang/3.0.0-beta1-RC0/
>
> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
>
> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
> improvements, and feature enhancements. Notable additions include the
> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
> shaded client, and HDFS erasure coding pluggable policy support.
>
> I've done the traditional testing of running a Pi job on a pseudo cluster.
> My +1 to start.
>
> We're working internally on getting this run through our integration test
> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.
>
> Best,
> Andrew
>
-- 



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[Andrew Wang]

Thanks for all the votes thus far! We've gotten the binding +1's to close
the release, though are there contributors who could kick the tires on
S3Guard and YARN TSv2 alpha2? These are the two new features merged since
alpha4, so it'd be good to get some coverage.



On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula 
wrote:

>
> Thanks Andrew.
>
> +1 (non binding)
>
> --Built from source
> --installed 3 node HA cluster
> --Verified shell commands and UI
> --Ran wordcount/pic jobs
>
>
>
>
> On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang 
> wrote:
>
>> Hi all,
>>
>> Let me start, as always, by thanking the many, many contributors who
>> helped
>> with this release! I've prepared an RC0 for 3.0.0-beta1:
>>
>> http://home.apache.org/~wang/3.0.0-beta1-RC0/
>>
>> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
>>
>> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
>> improvements, and feature enhancements. Notable additions include the
>> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
>> shaded client, and HDFS erasure coding pluggable policy support.
>>
>> I've done the traditional testing of running a Pi job on a pseudo cluster.
>> My +1 to start.
>>
>> We're working internally on getting this run through our integration test
>> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.
>>
>> Best,
>> Andrew
>>
> --
>
>
>
> --Brahma Reddy Battula
>

[jira] [Created] (HADOOP-14923) hadoop-azure has missing dependencies

[John Zhuge (JIRA)]

John Zhuge created HADOOP-14923:
---

 Summary: hadoop-azure has missing dependencies
 Key: HADOOP-14923
 URL: https://issues.apache.org/jira/browse/HADOOP-14923
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/azure
Affects Versions: 3.0.0-beta1
Reporter: John Zhuge
Priority: Minor


Saw these errors uncovered by dist-tools-hooks-maker during build:
{noformat}
ERROR: hadoop-azure has missing dependencies: 
jetty-util-ajax-9.3.19.v20170502.jar
{noformat}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14924) hadoop-azure-datalake has missing dependencies

[John Zhuge (JIRA)]

John Zhuge created HADOOP-14924:
---

 Summary: hadoop-azure-datalake has missing dependencies
 Key: HADOOP-14924
 URL: https://issues.apache.org/jira/browse/HADOOP-14924
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/adl
Affects Versions: 3.0.0-beta1
Reporter: John Zhuge
Priority: Minor


Saw these errors uncovered by dist-tools-hooks-maker during build:
{noformat}
ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
{noformat}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14925) hadoop-aliyun has missing dependencies

[John Zhuge (JIRA)]

John Zhuge created HADOOP-14925:
---

 Summary: hadoop-aliyun has missing dependencies
 Key: HADOOP-14925
 URL: https://issues.apache.org/jira/browse/HADOOP-14925
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/oss
Affects Versions: 3.0.0-beta1
Reporter: John Zhuge
Priority: Minor


Saw these errors uncovered by dist-tools-hooks-maker during build:
{noformat}
ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
{noformat}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[John Zhuge]

+1 (binding)

   - Verified checksums and signatures of all tarballs
   - Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
   - Verified cloud connectors:
  - All S3A integration tests
  - All ADL live unit tests
   - Deployed both binary and built source to a pseudo cluster, passed the
   following sanity tests in insecure, SSL, and SSL+Kerberos mode:
  - HDFS basic and ACL
  - DistCp basic
  - MapReduce wordcount (only failed in SSL+Kerberos mode for binary
  tarball, probably unrelated)
  - KMS and HttpFS basic
  - Balancer start/stop

Hit the following errors but they don't seem to be blocking:

== Missing dependencies during build ==

> ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> ERROR: hadoop-azure has missing dependencies: jetty-util-ajax-9.3.19.
> v20170502.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar


Filed HADOOP-14923, HADOOP-14924, and HADOOP-14925.

== Unit tests failed in Kerberos+SSL mode for KMS and HttpFs default HTTP
servlet /conf, /stacks, and /logLevel ==

One example below:

>Connecting to
> https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.HttpFSServer
>Exception in thread "main"
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Authentication failed, URL:
> https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.HttpFSServer&user.name=jzhuge,
> status: 403, message: GSSException: Failure unspecified at GSS-API level
> (Mechanism level: Request is a replay (34))


The /logLevel failure will affect command "hadoop daemonlog".


On Tue, Oct 3, 2017 at 10:56 AM, Andrew Wang 
wrote:

> Thanks for all the votes thus far! We've gotten the binding +1's to close
> the release, though are there contributors who could kick the tires on
> S3Guard and YARN TSv2 alpha2? These are the two new features merged since
> alpha4, so it'd be good to get some coverage.
>
>
>
> On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula 
> wrote:
>
> >
> > Thanks Andrew.
> >
> > +1 (non binding)
> >
> > --Built from source
> > --installed 3 node HA cluster
> > --Verified shell commands and UI
> > --Ran wordcount/pic jobs
> >
> >
> >
> >
> > On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang 
> > wrote:
> >
> >> Hi all,
> >>
> >> Let me start, as always, by thanking the many, many contributors who
> >> helped
> >> with this release! I've prepared an RC0 for 3.0.0-beta1:
> >>
> >> http://home.apache.org/~wang/3.0.0-beta1-RC0/
> >>
> >> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
> >>
> >> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
> >> improvements, and feature enhancements. Notable additions include the
> >> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
> >> shaded client, and HDFS erasure coding pluggable policy support.
> >>
> >> I've done the traditional testing of running a Pi job on a pseudo
> cluster.
> >> My +1 to start.
> >>
> >> We're working internally on getting this run through our integration
> test
> >> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.
> >>
> >> Best,
> >> Andrew
> >>
> > --
> >
> >
> >
> > --Brahma Reddy Battula
> >
>



-- 
John

[jira] [Resolved] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-14220.
-
   Resolution: Fixed
Fix Version/s: 2.9.0

> Enhance S3GuardTool with bucket-info and set-capacity commands, tests
> -
>
> Key: HADOOP-14220
> URL: https://issues.apache.org/jira/browse/HADOOP-14220
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, 
> HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch, 
> HADOOP-14220-013.patch, HADOOP-14220-013.patch, HADOOP-14220-014.patch, 
> HADOOP-14220-015.patch, HADOOP-14220-016.patch, 
> HADOOP-14220-branch-2-016.patch, HADOOP-14220-branch-2-017.patch, 
> HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, 
> HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, 
> HADOOP-14220-HADOOP-13345-005.patch
>
>
> Add a diagnostics command to s3guard which does whatever we need to diagnose 
> problems for a specific (named) s3a url. This is something which can be 
> attached to bug reports as well as used by developers.
> * Properties to log (with provenance attribute, which can track bucket 
> overrides: s3guard metastore setup, autocreate, capacity, 
> * table present/absent
> * # of keys in DDB table for that bucket?
> * any other stats?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[Eric Badger]

+1 (non-binding)

- Verified all checksums and signatures
- Built native from source on macOS 10.12.6 and RHEL 7.1
- Deployed a single node pseudo cluster
- Ran pi and sleep jobs
- Verified Docker was marked as experimental

Thanks,

Eric

On Tue, Oct 3, 2017 at 1:41 PM, John Zhuge  wrote:

> +1 (binding)
>
>- Verified checksums and signatures of all tarballs
>- Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
>- Verified cloud connectors:
>   - All S3A integration tests
>   - All ADL live unit tests
>- Deployed both binary and built source to a pseudo cluster, passed the
>following sanity tests in insecure, SSL, and SSL+Kerberos mode:
>   - HDFS basic and ACL
>   - DistCp basic
>   - MapReduce wordcount (only failed in SSL+Kerberos mode for binary
>   tarball, probably unrelated)
>   - KMS and HttpFS basic
>   - Balancer start/stop
>
> Hit the following errors but they don't seem to be blocking:
>
> == Missing dependencies during build ==
>
> > ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> > ERROR: hadoop-azure has missing dependencies: jetty-util-ajax-9.3.19.
> > v20170502.jar
> > ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> > ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
>
>
> Filed HADOOP-14923, HADOOP-14924, and HADOOP-14925.
>
> == Unit tests failed in Kerberos+SSL mode for KMS and HttpFs default HTTP
> servlet /conf, /stacks, and /logLevel ==
>
> One example below:
>
> >Connecting to
> > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.
> HttpFSServer
> >Exception in thread "main"
> > org.apache.hadoop.security.authentication.client.
> AuthenticationException:
> > Authentication failed, URL:
> > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.
> HttpFSServer&user.name=jzhuge,
> > status: 403, message: GSSException: Failure unspecified at GSS-API level
> > (Mechanism level: Request is a replay (34))
>
>
> The /logLevel failure will affect command "hadoop daemonlog".
>
>
> On Tue, Oct 3, 2017 at 10:56 AM, Andrew Wang 
> wrote:
>
> > Thanks for all the votes thus far! We've gotten the binding +1's to close
> > the release, though are there contributors who could kick the tires on
> > S3Guard and YARN TSv2 alpha2? These are the two new features merged since
> > alpha4, so it'd be good to get some coverage.
> >
> >
> >
> > On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula 
> > wrote:
> >
> > >
> > > Thanks Andrew.
> > >
> > > +1 (non binding)
> > >
> > > --Built from source
> > > --installed 3 node HA cluster
> > > --Verified shell commands and UI
> > > --Ran wordcount/pic jobs
> > >
> > >
> > >
> > >
> > > On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang 
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> Let me start, as always, by thanking the many, many contributors who
> > >> helped
> > >> with this release! I've prepared an RC0 for 3.0.0-beta1:
> > >>
> > >> http://home.apache.org/~wang/3.0.0-beta1-RC0/
> > >>
> > >> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
> > >>
> > >> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
> > >> improvements, and feature enhancements. Notable additions include the
> > >> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of
> the
> > >> shaded client, and HDFS erasure coding pluggable policy support.
> > >>
> > >> I've done the traditional testing of running a Pi job on a pseudo
> > cluster.
> > >> My +1 to start.
> > >>
> > >> We're working internally on getting this run through our integration
> > test
> > >> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's
> complete.
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > > --
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> > >
> >
>
>
>
> --
> John
>

[jira] [Created] (HADOOP-14926) Reconsider the default value of RPC timeout and document it

[Yufei Gu (JIRA)]

Yufei Gu created HADOOP-14926:
-

 Summary: Reconsider the default value of RPC timeout and document 
it
 Key: HADOOP-14926
 URL: https://issues.apache.org/jira/browse/HADOOP-14926
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common
Affects Versions: 3.0.0-alpha3, 2.8.1
Reporter: Yufei Gu


We use zero as the default value of RPC timeout, which mean we don't enforce 
any timeout aka infinity timeout. I think that *zero means infinite* is 
counter-intuitive and error-prone through some Java libs(e.g. Socket 
#setSoTimeout()) do that as well. Zero could be considered as a valid timeout 
value, while negative one isn't. If we use zero to represent infinite, which 
number could be used to represent zero timeout? I suggest use -1 as the default 
value to indicate infinite. 
We also need to document the default value and it means infinite timeout. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[Ajay Kumar]

+1 (non-binding)

- built from source
- deployed on single node cluster
- Basic hdfs operations 
- Run wordcount on a text file
Thanks,
Ajay


On 10/3/17, 1:04 PM, "Eric Badger"  wrote:

+1 (non-binding)

- Verified all checksums and signatures
- Built native from source on macOS 10.12.6 and RHEL 7.1
- Deployed a single node pseudo cluster
- Ran pi and sleep jobs
- Verified Docker was marked as experimental

Thanks,

Eric

On Tue, Oct 3, 2017 at 1:41 PM, John Zhuge  wrote:

> +1 (binding)
>
>- Verified checksums and signatures of all tarballs
>- Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
>- Verified cloud connectors:
>   - All S3A integration tests
>   - All ADL live unit tests
>- Deployed both binary and built source to a pseudo cluster, passed the
>following sanity tests in insecure, SSL, and SSL+Kerberos mode:
>   - HDFS basic and ACL
>   - DistCp basic
>   - MapReduce wordcount (only failed in SSL+Kerberos mode for binary
>   tarball, probably unrelated)
>   - KMS and HttpFS basic
>   - Balancer start/stop
>
> Hit the following errors but they don't seem to be blocking:
>
> == Missing dependencies during build ==
>
> > ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> > ERROR: hadoop-azure has missing dependencies: jetty-util-ajax-9.3.19.
> > v20170502.jar
> > ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> > ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
>
>
> Filed HADOOP-14923, HADOOP-14924, and HADOOP-14925.
>
> == Unit tests failed in Kerberos+SSL mode for KMS and HttpFs default HTTP
> servlet /conf, /stacks, and /logLevel ==
>
> One example below:
>
> >Connecting to
> > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.
> HttpFSServer
> >Exception in thread "main"
> > org.apache.hadoop.security.authentication.client.
> AuthenticationException:
> > Authentication failed, URL:
> > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.
> HttpFSServer&user.name=jzhuge,
> > status: 403, message: GSSException: Failure unspecified at GSS-API level
> > (Mechanism level: Request is a replay (34))
>
>
> The /logLevel failure will affect command "hadoop daemonlog".
>
>
> On Tue, Oct 3, 2017 at 10:56 AM, Andrew Wang 
> wrote:
>
> > Thanks for all the votes thus far! We've gotten the binding +1's to 
close
> > the release, though are there contributors who could kick the tires on
> > S3Guard and YARN TSv2 alpha2? These are the two new features merged 
since
> > alpha4, so it'd be good to get some coverage.
> >
> >
> >
> > On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula 
> > wrote:
> >
> > >
> > > Thanks Andrew.
> > >
> > > +1 (non binding)
> > >
> > > --Built from source
> > > --installed 3 node HA cluster
> > > --Verified shell commands and UI
> > > --Ran wordcount/pic jobs
> > >
> > >
> > >
> > >
> > > On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang 
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> Let me start, as always, by thanking the many, many contributors who
> > >> helped
> > >> with this release! I've prepared an RC0 for 3.0.0-beta1:
> > >>
> > >> http://home.apache.org/~wang/3.0.0-beta1-RC0/
> > >>
> > >> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
> > >>
> > >> beta1 contains 576 fixed JIRA issues comprising a number of bug 
fixes,
> > >> improvements, and feature enhancements. Notable additions include the
> > >> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of
> the
> > >> shaded client, and HDFS erasure coding pluggable policy support.
> > >>
> > >> I've done the traditional testing of running a Pi job on a pseudo
> > cluster.
> > >> My +1 to start.
> > >>
> > >> We're working internally on getting this run through our integration
> > test
> > >> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's
> complete.
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > > --
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> > >
> >
>
>
>
> --
> John
>

[jira] [Created] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

[Aaron Fabbri (JIRA)]

Aaron Fabbri created HADOOP-14927:
-

 Summary: ITestS3GuardTool failures in testDestroyNoBucket()
 Key: HADOOP-14927
 URL: https://issues.apache.org/jira/browse/HADOOP-14927
 Project: Hadoop Common
  Issue Type: Bug
  Components: s3
Affects Versions: 3.0.0-alpha3
Reporter: Aaron Fabbri
Priority: Minor


Hit this when testing for the Hadoop 3.0.0-beta1 RC0.

{noformat}
hadoop-3.0.0-beta1-src/hadoop-tools/hadoop-aws$ mvn clean verify 
-Dit.test="ITestS3GuardTool*" -Dtest=none -Ds3guard -Ddynamo
...
Failed tests: 
  ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:228 
Expected an exception, got 0
  ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:228 
Expected an exception, got 0
{noformat}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[Aaron Fabbri]

+1

Built from source.  Ran S3A integration tests in us-west-2 with S3Guard
(both Local and Dynamo metadatastore).

Everything worked fine except I hit one integration test failure.  It is a
minor test issue IMO and I've filed HADOOP-14927

Failed tests:

ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:228
Expected an exception, got 0
  ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:228
Expected an exception, got 0



On Tue, Oct 3, 2017 at 2:45 PM, Ajay Kumar 
wrote:

> +1 (non-binding)
>
> - built from source
> - deployed on single node cluster
> - Basic hdfs operations
> - Run wordcount on a text file
> Thanks,
> Ajay
>
>
> On 10/3/17, 1:04 PM, "Eric Badger"  wrote:
>
> +1 (non-binding)
>
> - Verified all checksums and signatures
> - Built native from source on macOS 10.12.6 and RHEL 7.1
> - Deployed a single node pseudo cluster
> - Ran pi and sleep jobs
> - Verified Docker was marked as experimental
>
> Thanks,
>
> Eric
>
> On Tue, Oct 3, 2017 at 1:41 PM, John Zhuge 
> wrote:
>
> > +1 (binding)
> >
> >- Verified checksums and signatures of all tarballs
> >- Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
> >- Verified cloud connectors:
> >   - All S3A integration tests
> >   - All ADL live unit tests
> >- Deployed both binary and built source to a pseudo cluster,
> passed the
> >following sanity tests in insecure, SSL, and SSL+Kerberos mode:
> >   - HDFS basic and ACL
> >   - DistCp basic
> >   - MapReduce wordcount (only failed in SSL+Kerberos mode for
> binary
> >   tarball, probably unrelated)
> >   - KMS and HttpFS basic
> >   - Balancer start/stop
> >
> > Hit the following errors but they don't seem to be blocking:
> >
> > == Missing dependencies during build ==
> >
> > > ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> > > ERROR: hadoop-azure has missing dependencies:
> jetty-util-ajax-9.3.19.
> > > v20170502.jar
> > > ERROR: hadoop-azure-datalake has missing dependencies:
> okhttp-2.4.0.jar
> > > ERROR: hadoop-azure-datalake has missing dependencies:
> okio-1.4.0.jar
> >
> >
> > Filed HADOOP-14923, HADOOP-14924, and HADOOP-14925.
> >
> > == Unit tests failed in Kerberos+SSL mode for KMS and HttpFs default
> HTTP
> > servlet /conf, /stacks, and /logLevel ==
> >
> > One example below:
> >
> > >Connecting to
> > > https://localhost:14000/logLevel?log=org.apache.
> hadoop.fs.http.server.
> > HttpFSServer
> > >Exception in thread "main"
> > > org.apache.hadoop.security.authentication.client.
> > AuthenticationException:
> > > Authentication failed, URL:
> > > https://localhost:14000/logLevel?log=org.apache.
> hadoop.fs.http.server.
> > HttpFSServer&user.name=jzhuge,
> > > status: 403, message: GSSException: Failure unspecified at GSS-API
> level
> > > (Mechanism level: Request is a replay (34))
> >
> >
> > The /logLevel failure will affect command "hadoop daemonlog".
> >
> >
> > On Tue, Oct 3, 2017 at 10:56 AM, Andrew Wang <
> andrew.w...@cloudera.com>
> > wrote:
> >
> > > Thanks for all the votes thus far! We've gotten the binding +1's
> to close
> > > the release, though are there contributors who could kick the
> tires on
> > > S3Guard and YARN TSv2 alpha2? These are the two new features
> merged since
> > > alpha4, so it'd be good to get some coverage.
> > >
> > >
> > >
> > > On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula <
> bra...@apache.org>
> > > wrote:
> > >
> > > >
> > > > Thanks Andrew.
> > > >
> > > > +1 (non binding)
> > > >
> > > > --Built from source
> > > > --installed 3 node HA cluster
> > > > --Verified shell commands and UI
> > > > --Ran wordcount/pic jobs
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang <
> andrew.w...@cloudera.com>
> > > > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> Let me start, as always, by thanking the many, many
> contributors who
> > > >> helped
> > > >> with this release! I've prepared an RC0 for 3.0.0-beta1:
> > > >>
> > > >> http://home.apache.org/~wang/3.0.0-beta1-RC0/
> > > >>
> > > >> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
> > > >>
> > > >> beta1 contains 576 fixed JIRA issues comprising a number of bug
> fixes,
> > > >> improvements, and feature enhancements. Notable additions
> include the
> > > >> addition of YARN Timeline Service v2 alpha2, S3Guard,
> completion of
> > the
> > > >> shaded client, and HDFS erasure coding pluggable policy support.
> > > >>
> > > >> I've done the traditional testing of running a Pi job on a
> 

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[Andrew Wang]

Thanks everyone for voting! With 4 binding +1s and 7 non-binding +1s, the
vote passes.

I'll get started on pushing out the release.

Best,
Andrew

On Tue, Oct 3, 2017 at 3:45 PM, Aaron Fabbri  wrote:

> +1
>
> Built from source.  Ran S3A integration tests in us-west-2 with S3Guard
> (both Local and Dynamo metadatastore).
>
> Everything worked fine except I hit one integration test failure.  It is a
> minor test issue IMO and I've filed HADOOP-14927
>
> Failed tests:
>   ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:228
> Expected an exception, got 0
>   ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:228
> Expected an exception, got 0
>
>
>
> On Tue, Oct 3, 2017 at 2:45 PM, Ajay Kumar 
> wrote:
>
>> +1 (non-binding)
>>
>> - built from source
>> - deployed on single node cluster
>> - Basic hdfs operations
>> - Run wordcount on a text file
>> Thanks,
>> Ajay
>>
>>
>> On 10/3/17, 1:04 PM, "Eric Badger"  wrote:
>>
>> +1 (non-binding)
>>
>> - Verified all checksums and signatures
>> - Built native from source on macOS 10.12.6 and RHEL 7.1
>> - Deployed a single node pseudo cluster
>> - Ran pi and sleep jobs
>> - Verified Docker was marked as experimental
>>
>> Thanks,
>>
>> Eric
>>
>> On Tue, Oct 3, 2017 at 1:41 PM, John Zhuge 
>> wrote:
>>
>> > +1 (binding)
>> >
>> >- Verified checksums and signatures of all tarballs
>> >- Built source with native, Java 1.8.0_131-b11 on Mac OS X
>> 10.12.6
>> >- Verified cloud connectors:
>> >   - All S3A integration tests
>> >   - All ADL live unit tests
>> >- Deployed both binary and built source to a pseudo cluster,
>> passed the
>> >following sanity tests in insecure, SSL, and SSL+Kerberos mode:
>> >   - HDFS basic and ACL
>> >   - DistCp basic
>> >   - MapReduce wordcount (only failed in SSL+Kerberos mode for
>> binary
>> >   tarball, probably unrelated)
>> >   - KMS and HttpFS basic
>> >   - Balancer start/stop
>> >
>> > Hit the following errors but they don't seem to be blocking:
>> >
>> > == Missing dependencies during build ==
>> >
>> > > ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
>> > > ERROR: hadoop-azure has missing dependencies:
>> jetty-util-ajax-9.3.19.
>> > > v20170502.jar
>> > > ERROR: hadoop-azure-datalake has missing dependencies:
>> okhttp-2.4.0.jar
>> > > ERROR: hadoop-azure-datalake has missing dependencies:
>> okio-1.4.0.jar
>> >
>> >
>> > Filed HADOOP-14923, HADOOP-14924, and HADOOP-14925.
>> >
>> > == Unit tests failed in Kerberos+SSL mode for KMS and HttpFs
>> default HTTP
>> > servlet /conf, /stacks, and /logLevel ==
>> >
>> > One example below:
>> >
>> > >Connecting to
>> > > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.
>> http.server.
>> > HttpFSServer
>> > >Exception in thread "main"
>> > > org.apache.hadoop.security.authentication.client.
>> > AuthenticationException:
>> > > Authentication failed, URL:
>> > > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.
>> http.server.
>> > HttpFSServer&user.name=jzhuge,
>> > > status: 403, message: GSSException: Failure unspecified at
>> GSS-API level
>> > > (Mechanism level: Request is a replay (34))
>> >
>> >
>> > The /logLevel failure will affect command "hadoop daemonlog".
>> >
>> >
>> > On Tue, Oct 3, 2017 at 10:56 AM, Andrew Wang <
>> andrew.w...@cloudera.com>
>> > wrote:
>> >
>> > > Thanks for all the votes thus far! We've gotten the binding +1's
>> to close
>> > > the release, though are there contributors who could kick the
>> tires on
>> > > S3Guard and YARN TSv2 alpha2? These are the two new features
>> merged since
>> > > alpha4, so it'd be good to get some coverage.
>> > >
>> > >
>> > >
>> > > On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula <
>> bra...@apache.org>
>> > > wrote:
>> > >
>> > > >
>> > > > Thanks Andrew.
>> > > >
>> > > > +1 (non binding)
>> > > >
>> > > > --Built from source
>> > > > --installed 3 node HA cluster
>> > > > --Verified shell commands and UI
>> > > > --Ran wordcount/pic jobs
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang <
>> andrew.w...@cloudera.com>
>> > > > wrote:
>> > > >
>> > > >> Hi all,
>> > > >>
>> > > >> Let me start, as always, by thanking the many, many
>> contributors who
>> > > >> helped
>> > > >> with this release! I've prepared an RC0 for 3.0.0-beta1:
>> > > >>
>> > > >> http://home.apache.org/~wang/3.0.0-beta1-RC0/
>> > > >>
>> > > >> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
>> > > >>
>> > > >> beta1 contains 576 fixed JIRA issues comprising