Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

2017-10-03 Thread Andrew Wang 
Thanks everyone for voting! With 4 binding +1s and 7 non-binding +1s, the
vote passes.

I'll get started on pushing out the release.

Best,
Andrew

On Tue, Oct 3, 2017 at 3:45 PM, Aaron Fabbri  wrote:

> +1
>
> Built from source.  Ran S3A integration tests in us-west-2 with S3Guard
> (both Local and Dynamo metadatastore).
>
> Everything worked fine except I hit one integration test failure.  It is a
> minor test issue IMO and I've filed HADOOP-14927
>
> Failed tests:
>   ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:228
> Expected an exception, got 0
>   ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:228
> Expected an exception, got 0
>
>
>
> On Tue, Oct 3, 2017 at 2:45 PM, Ajay Kumar 
> wrote:
>
>> +1 (non-binding)
>>
>> - built from source
>> - deployed on single node cluster
>> - Basic hdfs operations
>> - Run wordcount on a text file
>> Thanks,
>> Ajay
>>
>>
>> On 10/3/17, 1:04 PM, "Eric Badger"  wrote:
>>
>> +1 (non-binding)
>>
>> - Verified all checksums and signatures
>> - Built native from source on macOS 10.12.6 and RHEL 7.1
>> - Deployed a single node pseudo cluster
>> - Ran pi and sleep jobs
>> - Verified Docker was marked as experimental
>>
>> Thanks,
>>
>> Eric
>>
>> On Tue, Oct 3, 2017 at 1:41 PM, John Zhuge 
>> wrote:
>>
>> > +1 (binding)
>> >
>> >- Verified checksums and signatures of all tarballs
>> >- Built source with native, Java 1.8.0_131-b11 on Mac OS X
>> 10.12.6
>> >- Verified cloud connectors:
>> >   - All S3A integration tests
>> >   - All ADL live unit tests
>> >- Deployed both binary and built source to a pseudo cluster,
>> passed the
>> >following sanity tests in insecure, SSL, and SSL+Kerberos mode:
>> >   - HDFS basic and ACL
>> >   - DistCp basic
>> >   - MapReduce wordcount (only failed in SSL+Kerberos mode for
>> binary
>> >   tarball, probably unrelated)
>> >   - KMS and HttpFS basic
>> >   - Balancer start/stop
>> >
>> > Hit the following errors but they don't seem to be blocking:
>> >
>> > == Missing dependencies during build ==
>> >
>> > > ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
>> > > ERROR: hadoop-azure has missing dependencies:
>> jetty-util-ajax-9.3.19.
>> > > v20170502.jar
>> > > ERROR: hadoop-azure-datalake has missing dependencies:
>> okhttp-2.4.0.jar
>> > > ERROR: hadoop-azure-datalake has missing dependencies:
>> okio-1.4.0.jar
>> >
>> >
>> > Filed HADOOP-14923, HADOOP-14924, and HADOOP-14925.
>> >
>> > == Unit tests failed in Kerberos+SSL mode for KMS and HttpFs
>> default HTTP
>> > servlet /conf, /stacks, and /logLevel ==
>> >
>> > One example below:
>> >
>> > >Connecting to
>> > > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.
>> http.server.
>> > HttpFSServer
>> > >Exception in thread "main"
>> > > org.apache.hadoop.security.authentication.client.
>> > AuthenticationException:
>> > > Authentication failed, URL:
>> > > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.
>> http.server.
>> > HttpFSServer=jzhuge,
>> > > status: 403, message: GSSException: Failure unspecified at
>> GSS-API level
>> > > (Mechanism level: Request is a replay (34))
>> >
>> >
>> > The /logLevel failure will affect command "hadoop daemonlog".
>> >
>> >
>> > On Tue, Oct 3, 2017 at 10:56 AM, Andrew Wang <
>> andrew.w...@cloudera.com>
>> > wrote:
>> >
>> > > Thanks for all the votes thus far! We've gotten the binding +1's
>> to close
>> > > the release, though are there contributors who could kick the
>> tires on
>> > > S3Guard and YARN TSv2 alpha2? These are the two new features
>> merged since
>> > > alpha4, so it'd be good to get some coverage.
>> > >
>> > >
>> > >
>> > > On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula <
>> bra...@apache.org>
>> > > wrote:
>> > >
>> > > >
>> > > > Thanks Andrew.
>> > > >
>> > > > +1 (non binding)
>> > > >
>> > > > --Built from source
>> > > > --installed 3 node HA cluster
>> > > > --Verified shell commands and UI
>> > > > --Ran wordcount/pic jobs
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang <
>> andrew.w...@cloudera.com>
>> > > > wrote:
>> > > >
>> > > >> Hi all,
>> > > >>
>> > > >> Let me start, as always, by thanking the many, many
>> contributors who
>> > > >> helped
>> > > >> with this release! I've prepared an RC0 for 3.0.0-beta1:
>> > > >>
>> > > >> http://home.apache.org/~wang/3.0.0-beta1-RC0/
>> > > >>
>> > > >> This vote will run five days, ending on Nov 3rd at

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

2017-10-03 Thread Aaron Fabbri 
+1

Built from source.  Ran S3A integration tests in us-west-2 with S3Guard
(both Local and Dynamo metadatastore).

Everything worked fine except I hit one integration test failure.  It is a
minor test issue IMO and I've filed HADOOP-14927

Failed tests:

ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:228
Expected an exception, got 0
  ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:228
Expected an exception, got 0



On Tue, Oct 3, 2017 at 2:45 PM, Ajay Kumar 
wrote:

> +1 (non-binding)
>
> - built from source
> - deployed on single node cluster
> - Basic hdfs operations
> - Run wordcount on a text file
> Thanks,
> Ajay
>
>
> On 10/3/17, 1:04 PM, "Eric Badger"  wrote:
>
> +1 (non-binding)
>
> - Verified all checksums and signatures
> - Built native from source on macOS 10.12.6 and RHEL 7.1
> - Deployed a single node pseudo cluster
> - Ran pi and sleep jobs
> - Verified Docker was marked as experimental
>
> Thanks,
>
> Eric
>
> On Tue, Oct 3, 2017 at 1:41 PM, John Zhuge 
> wrote:
>
> > +1 (binding)
> >
> >- Verified checksums and signatures of all tarballs
> >- Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
> >- Verified cloud connectors:
> >   - All S3A integration tests
> >   - All ADL live unit tests
> >- Deployed both binary and built source to a pseudo cluster,
> passed the
> >following sanity tests in insecure, SSL, and SSL+Kerberos mode:
> >   - HDFS basic and ACL
> >   - DistCp basic
> >   - MapReduce wordcount (only failed in SSL+Kerberos mode for
> binary
> >   tarball, probably unrelated)
> >   - KMS and HttpFS basic
> >   - Balancer start/stop
> >
> > Hit the following errors but they don't seem to be blocking:
> >
> > == Missing dependencies during build ==
> >
> > > ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> > > ERROR: hadoop-azure has missing dependencies:
> jetty-util-ajax-9.3.19.
> > > v20170502.jar
> > > ERROR: hadoop-azure-datalake has missing dependencies:
> okhttp-2.4.0.jar
> > > ERROR: hadoop-azure-datalake has missing dependencies:
> okio-1.4.0.jar
> >
> >
> > Filed HADOOP-14923, HADOOP-14924, and HADOOP-14925.
> >
> > == Unit tests failed in Kerberos+SSL mode for KMS and HttpFs default
> HTTP
> > servlet /conf, /stacks, and /logLevel ==
> >
> > One example below:
> >
> > >Connecting to
> > > https://localhost:14000/logLevel?log=org.apache.
> hadoop.fs.http.server.
> > HttpFSServer
> > >Exception in thread "main"
> > > org.apache.hadoop.security.authentication.client.
> > AuthenticationException:
> > > Authentication failed, URL:
> > > https://localhost:14000/logLevel?log=org.apache.
> hadoop.fs.http.server.
> > HttpFSServer=jzhuge,
> > > status: 403, message: GSSException: Failure unspecified at GSS-API
> level
> > > (Mechanism level: Request is a replay (34))
> >
> >
> > The /logLevel failure will affect command "hadoop daemonlog".
> >
> >
> > On Tue, Oct 3, 2017 at 10:56 AM, Andrew Wang <
> andrew.w...@cloudera.com>
> > wrote:
> >
> > > Thanks for all the votes thus far! We've gotten the binding +1's
> to close
> > > the release, though are there contributors who could kick the
> tires on
> > > S3Guard and YARN TSv2 alpha2? These are the two new features
> merged since
> > > alpha4, so it'd be good to get some coverage.
> > >
> > >
> > >
> > > On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula <
> bra...@apache.org>
> > > wrote:
> > >
> > > >
> > > > Thanks Andrew.
> > > >
> > > > +1 (non binding)
> > > >
> > > > --Built from source
> > > > --installed 3 node HA cluster
> > > > --Verified shell commands and UI
> > > > --Ran wordcount/pic jobs
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang <
> andrew.w...@cloudera.com>
> > > > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> Let me start, as always, by thanking the many, many
> contributors who
> > > >> helped
> > > >> with this release! I've prepared an RC0 for 3.0.0-beta1:
> > > >>
> > > >> http://home.apache.org/~wang/3.0.0-beta1-RC0/
> > > >>
> > > >> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
> > > >>
> > > >> beta1 contains 576 fixed JIRA issues comprising a number of bug
> fixes,
> > > >> improvements, and feature enhancements. Notable additions
> include the
> > > >> addition of YARN Timeline Service v2 alpha2, S3Guard,
> completion of
> > the
> > > >> shaded client, and HDFS erasure coding pluggable policy support.
> > > >>
> >

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

2017-10-03 Thread Ajay Kumar 
+1 (non-binding)

- built from source
- deployed on single node cluster
- Basic hdfs operations 
- Run wordcount on a text file
Thanks,
Ajay


On 10/3/17, 1:04 PM, "Eric Badger"  wrote:

+1 (non-binding)

- Verified all checksums and signatures
- Built native from source on macOS 10.12.6 and RHEL 7.1
- Deployed a single node pseudo cluster
- Ran pi and sleep jobs
- Verified Docker was marked as experimental

Thanks,

Eric

On Tue, Oct 3, 2017 at 1:41 PM, John Zhuge  wrote:

> +1 (binding)
>
>- Verified checksums and signatures of all tarballs
>- Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
>- Verified cloud connectors:
>   - All S3A integration tests
>   - All ADL live unit tests
>- Deployed both binary and built source to a pseudo cluster, passed the
>following sanity tests in insecure, SSL, and SSL+Kerberos mode:
>   - HDFS basic and ACL
>   - DistCp basic
>   - MapReduce wordcount (only failed in SSL+Kerberos mode for binary
>   tarball, probably unrelated)
>   - KMS and HttpFS basic
>   - Balancer start/stop
>
> Hit the following errors but they don't seem to be blocking:
>
> == Missing dependencies during build ==
>
> > ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> > ERROR: hadoop-azure has missing dependencies: jetty-util-ajax-9.3.19.
> > v20170502.jar
> > ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> > ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
>
>
> Filed HADOOP-14923, HADOOP-14924, and HADOOP-14925.
>
> == Unit tests failed in Kerberos+SSL mode for KMS and HttpFs default HTTP
> servlet /conf, /stacks, and /logLevel ==
>
> One example below:
>
> >Connecting to
> > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.
> HttpFSServer
> >Exception in thread "main"
> > org.apache.hadoop.security.authentication.client.
> AuthenticationException:
> > Authentication failed, URL:
> > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.
> HttpFSServer=jzhuge,
> > status: 403, message: GSSException: Failure unspecified at GSS-API level
> > (Mechanism level: Request is a replay (34))
>
>
> The /logLevel failure will affect command "hadoop daemonlog".
>
>
> On Tue, Oct 3, 2017 at 10:56 AM, Andrew Wang 
> wrote:
>
> > Thanks for all the votes thus far! We've gotten the binding +1's to 
close
> > the release, though are there contributors who could kick the tires on
> > S3Guard and YARN TSv2 alpha2? These are the two new features merged 
since
> > alpha4, so it'd be good to get some coverage.
> >
> >
> >
> > On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula 
> > wrote:
> >
> > >
> > > Thanks Andrew.
> > >
> > > +1 (non binding)
> > >
> > > --Built from source
> > > --installed 3 node HA cluster
> > > --Verified shell commands and UI
> > > --Ran wordcount/pic jobs
> > >
> > >
> > >
> > >
> > > On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang 
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> Let me start, as always, by thanking the many, many contributors who
> > >> helped
> > >> with this release! I've prepared an RC0 for 3.0.0-beta1:
> > >>
> > >> http://home.apache.org/~wang/3.0.0-beta1-RC0/
> > >>
> > >> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
> > >>
> > >> beta1 contains 576 fixed JIRA issues comprising a number of bug 
fixes,
> > >> improvements, and feature enhancements. Notable additions include the
> > >> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of
> the
> > >> shaded client, and HDFS erasure coding pluggable policy support.
> > >>
> > >> I've done the traditional testing of running a Pi job on a pseudo
> > cluster.
> > >> My +1 to start.
> > >>
> > >> We're working internally on getting this run through our integration
> > test
> > >> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's
> complete.
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > > --
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> > >
> >
>
>
>
> --
> John
>

[jira] [Created] (HDFS-12586) EZ createZone returns IllegalArgumentException when using protocol in path

2017-10-03 Thread Rushabh S Shah (JIRA) 
Rushabh S Shah created HDFS-12586:
-

 Summary: EZ createZone returns IllegalArgumentException when using 
protocol in path
 Key: HDFS-12586
 URL: https://issues.apache.org/jira/browse/HDFS-12586
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Rushabh S Shah
Assignee: Rushabh S Shah


When trying to create an EZ and sending protocol (hdfs://) as part of the path, 
-createZone reports an IllegalArgumentException.
IllegalArgumentException: hdfs:///tmp/fooez1 is not the root 
of an encryption zone. Do you mean /tmp/fooez1?

Here's a sequence:
1. mkdir the path
bash-4.1$ hadoop fs -mkdir /tmp/fooez1
2. try to make EZ using hdfs protocol, and get error
hdfs crypto -createZone -keyName key1 -path 
hdfs:///tmp/fooez1/
IllegalArgumentException: hdfs:///tmp/fooez1 is not the root 
of an encryption zone. Do you mean /tmp/fooez1?

It fails while provisioning trash for ez root directory.
The relevant chunk of code.
{code:title=HdfsAdmin.java|borderStyle=solid}
private void provisionEZTrash(Path path) throws IOException {
   ...
   ...
String ezPath = ez.getPath();
if (!path.toString().equals(ezPath)) {
  throw new IllegalArgumentException(path + " is not the root of an " +
  "encryption zone. Do you mean " + ez.getPath() + "?");
}
{code}
It is comparing the {{supplied path}} with path component of 
{{EncryptionZone#path}} which doesn't contain scheme and authority.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

2017-10-03 Thread Eric Badger 
+1 (non-binding)

- Verified all checksums and signatures
- Built native from source on macOS 10.12.6 and RHEL 7.1
- Deployed a single node pseudo cluster
- Ran pi and sleep jobs
- Verified Docker was marked as experimental

Thanks,

Eric

On Tue, Oct 3, 2017 at 1:41 PM, John Zhuge  wrote:

> +1 (binding)
>
>- Verified checksums and signatures of all tarballs
>- Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
>- Verified cloud connectors:
>   - All S3A integration tests
>   - All ADL live unit tests
>- Deployed both binary and built source to a pseudo cluster, passed the
>following sanity tests in insecure, SSL, and SSL+Kerberos mode:
>   - HDFS basic and ACL
>   - DistCp basic
>   - MapReduce wordcount (only failed in SSL+Kerberos mode for binary
>   tarball, probably unrelated)
>   - KMS and HttpFS basic
>   - Balancer start/stop
>
> Hit the following errors but they don't seem to be blocking:
>
> == Missing dependencies during build ==
>
> > ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> > ERROR: hadoop-azure has missing dependencies: jetty-util-ajax-9.3.19.
> > v20170502.jar
> > ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> > ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
>
>
> Filed HADOOP-14923, HADOOP-14924, and HADOOP-14925.
>
> == Unit tests failed in Kerberos+SSL mode for KMS and HttpFs default HTTP
> servlet /conf, /stacks, and /logLevel ==
>
> One example below:
>
> >Connecting to
> > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.
> HttpFSServer
> >Exception in thread "main"
> > org.apache.hadoop.security.authentication.client.
> AuthenticationException:
> > Authentication failed, URL:
> > https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.
> HttpFSServer=jzhuge,
> > status: 403, message: GSSException: Failure unspecified at GSS-API level
> > (Mechanism level: Request is a replay (34))
>
>
> The /logLevel failure will affect command "hadoop daemonlog".
>
>
> On Tue, Oct 3, 2017 at 10:56 AM, Andrew Wang 
> wrote:
>
> > Thanks for all the votes thus far! We've gotten the binding +1's to close
> > the release, though are there contributors who could kick the tires on
> > S3Guard and YARN TSv2 alpha2? These are the two new features merged since
> > alpha4, so it'd be good to get some coverage.
> >
> >
> >
> > On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula 
> > wrote:
> >
> > >
> > > Thanks Andrew.
> > >
> > > +1 (non binding)
> > >
> > > --Built from source
> > > --installed 3 node HA cluster
> > > --Verified shell commands and UI
> > > --Ran wordcount/pic jobs
> > >
> > >
> > >
> > >
> > > On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang 
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> Let me start, as always, by thanking the many, many contributors who
> > >> helped
> > >> with this release! I've prepared an RC0 for 3.0.0-beta1:
> > >>
> > >> http://home.apache.org/~wang/3.0.0-beta1-RC0/
> > >>
> > >> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
> > >>
> > >> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
> > >> improvements, and feature enhancements. Notable additions include the
> > >> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of
> the
> > >> shaded client, and HDFS erasure coding pluggable policy support.
> > >>
> > >> I've done the traditional testing of running a Pi job on a pseudo
> > cluster.
> > >> My +1 to start.
> > >>
> > >> We're working internally on getting this run through our integration
> > test
> > >> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's
> complete.
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > > --
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> > >
> >
>
>
>
> --
> John
>

[jira] [Created] (HDFS-12585) Add description for each config in Ozone config UI

2017-10-03 Thread Ajay Kumar (JIRA) 
Ajay Kumar created HDFS-12585:
-

 Summary: Add description for each config in Ozone config UI
 Key: HDFS-12585
 URL: https://issues.apache.org/jira/browse/HDFS-12585
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: HDFS-7240
Reporter: Ajay Kumar
Assignee: Ajay Kumar
 Fix For: HDFS-7240


Add description for each config in Ozone config UI



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12442) WebHdfsFileSystem#getFileBlockLocations will always return BlockLocation#corrupt as false

2017-10-03 Thread Andrew Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang resolved HDFS-12442.

Resolution: Invalid

> WebHdfsFileSystem#getFileBlockLocations will always return 
> BlockLocation#corrupt as false
> -
>
> Key: HDFS-12442
> URL: https://issues.apache.org/jira/browse/HDFS-12442
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Critical
> Attachments: HDFS-12442-1.patch
>
>
> Was going through {{JsonUtilClient#toBlockLocation}} code.
> Below is the relevant code snippet.
> {code:title=JsonUtilClient.java|borderStyle=solid}
>  /** Convert a Json map to BlockLocation. **/
>   static BlockLocation toBlockLocation(Map m)
>   throws IOException{
> ...
> ...  
> boolean corrupt = Boolean.
> getBoolean(m.get("corrupt").toString());
> ...
> ...
>   }
> {code}
> According to java docs for {{Boolean#getBoolean}}
> {noformat}
> Returns true if and only if the system property named by the argument exists 
> and is equal to the string "true". 
> {noformat}
> I assume, the map value for key {{corrupt}} will be populated with either 
> {{true}} or {{false}}.
> On the client side, {{Boolean#getBoolean}} will look for system property for 
> true or false.
> So it will always return false unless the system property is set for true or 
> false.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-12442) WebHdfsFileSystem#getFileBlockLocations will always return BlockLocation#corrupt as false

2017-10-03 Thread Andrew Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang reopened HDFS-12442:


> WebHdfsFileSystem#getFileBlockLocations will always return 
> BlockLocation#corrupt as false
> -
>
> Key: HDFS-12442
> URL: https://issues.apache.org/jira/browse/HDFS-12442
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Critical
> Attachments: HDFS-12442-1.patch
>
>
> Was going through {{JsonUtilClient#toBlockLocation}} code.
> Below is the relevant code snippet.
> {code:title=JsonUtilClient.java|borderStyle=solid}
>  /** Convert a Json map to BlockLocation. **/
>   static BlockLocation toBlockLocation(Map m)
>   throws IOException{
> ...
> ...  
> boolean corrupt = Boolean.
> getBoolean(m.get("corrupt").toString());
> ...
> ...
>   }
> {code}
> According to java docs for {{Boolean#getBoolean}}
> {noformat}
> Returns true if and only if the system property named by the argument exists 
> and is equal to the string "true". 
> {noformat}
> I assume, the map value for key {{corrupt}} will be populated with either 
> {{true}} or {{false}}.
> On the client side, {{Boolean#getBoolean}} will look for system property for 
> true or false.
> So it will always return false unless the system property is set for true or 
> false.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12584) [READ] Fix errors in image generation tool from latest rebase

2017-10-03 Thread Virajith Jalaparti (JIRA) 
Virajith Jalaparti created HDFS-12584:
-

 Summary: [READ] Fix errors in image generation tool from latest 
rebase
 Key: HDFS-12584
 URL: https://issues.apache.org/jira/browse/HDFS-12584
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Virajith Jalaparti


Fix compile errors, from the latest rebase, in FSImage generation tool



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

2017-10-03 Thread John Zhuge 
+1 (binding)

   - Verified checksums and signatures of all tarballs
   - Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
   - Verified cloud connectors:
  - All S3A integration tests
  - All ADL live unit tests
   - Deployed both binary and built source to a pseudo cluster, passed the
   following sanity tests in insecure, SSL, and SSL+Kerberos mode:
  - HDFS basic and ACL
  - DistCp basic
  - MapReduce wordcount (only failed in SSL+Kerberos mode for binary
  tarball, probably unrelated)
  - KMS and HttpFS basic
  - Balancer start/stop

Hit the following errors but they don't seem to be blocking:

== Missing dependencies during build ==

> ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> ERROR: hadoop-azure has missing dependencies: jetty-util-ajax-9.3.19.
> v20170502.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar


Filed HADOOP-14923, HADOOP-14924, and HADOOP-14925.

== Unit tests failed in Kerberos+SSL mode for KMS and HttpFs default HTTP
servlet /conf, /stacks, and /logLevel ==

One example below:

>Connecting to
> https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.HttpFSServer
>Exception in thread "main"
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Authentication failed, URL:
> https://localhost:14000/logLevel?log=org.apache.hadoop.fs.http.server.HttpFSServer=jzhuge,
> status: 403, message: GSSException: Failure unspecified at GSS-API level
> (Mechanism level: Request is a replay (34))


The /logLevel failure will affect command "hadoop daemonlog".


On Tue, Oct 3, 2017 at 10:56 AM, Andrew Wang 
wrote:

> Thanks for all the votes thus far! We've gotten the binding +1's to close
> the release, though are there contributors who could kick the tires on
> S3Guard and YARN TSv2 alpha2? These are the two new features merged since
> alpha4, so it'd be good to get some coverage.
>
>
>
> On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula 
> wrote:
>
> >
> > Thanks Andrew.
> >
> > +1 (non binding)
> >
> > --Built from source
> > --installed 3 node HA cluster
> > --Verified shell commands and UI
> > --Ran wordcount/pic jobs
> >
> >
> >
> >
> > On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang 
> > wrote:
> >
> >> Hi all,
> >>
> >> Let me start, as always, by thanking the many, many contributors who
> >> helped
> >> with this release! I've prepared an RC0 for 3.0.0-beta1:
> >>
> >> http://home.apache.org/~wang/3.0.0-beta1-RC0/
> >>
> >> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
> >>
> >> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
> >> improvements, and feature enhancements. Notable additions include the
> >> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
> >> shaded client, and HDFS erasure coding pluggable policy support.
> >>
> >> I've done the traditional testing of running a Pi job on a pseudo
> cluster.
> >> My +1 to start.
> >>
> >> We're working internally on getting this run through our integration
> test
> >> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.
> >>
> >> Best,
> >> Andrew
> >>
> > --
> >
> >
> >
> > --Brahma Reddy Battula
> >
>



-- 
John

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

2017-10-03 Thread Andrew Wang 
Thanks for all the votes thus far! We've gotten the binding +1's to close
the release, though are there contributors who could kick the tires on
S3Guard and YARN TSv2 alpha2? These are the two new features merged since
alpha4, so it'd be good to get some coverage.



On Tue, Oct 3, 2017 at 9:45 AM, Brahma Reddy Battula 
wrote:

>
> Thanks Andrew.
>
> +1 (non binding)
>
> --Built from source
> --installed 3 node HA cluster
> --Verified shell commands and UI
> --Ran wordcount/pic jobs
>
>
>
>
> On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang 
> wrote:
>
>> Hi all,
>>
>> Let me start, as always, by thanking the many, many contributors who
>> helped
>> with this release! I've prepared an RC0 for 3.0.0-beta1:
>>
>> http://home.apache.org/~wang/3.0.0-beta1-RC0/
>>
>> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
>>
>> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
>> improvements, and feature enhancements. Notable additions include the
>> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
>> shaded client, and HDFS erasure coding pluggable policy support.
>>
>> I've done the traditional testing of running a Pi job on a pseudo cluster.
>> My +1 to start.
>>
>> We're working internally on getting this run through our integration test
>> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.
>>
>> Best,
>> Andrew
>>
> --
>
>
>
> --Brahma Reddy Battula
>

Re: [VOTE] Merge Router-Based Federation (HDFS-10467) branch into trunk/branch-3

2017-10-03 Thread Brahma Reddy Battula 
Thanks Inigo.

+1 (binding)

Nice feature.Involved in reviewing some jiras.

On Sat, 30 Sep 2017 at 12:29 AM, Iñigo Goiri  wrote:

> Hi all,
>
> Given that 3.0-beta1 is already cut, I’d like to call a vote for merging
> Router-Based Federation (HDFS-10467) to trunk and branch-3.
>
> The vote will run for 7 days as usual.
>
>
>
> We started the discussion about merging HDFS-10467 a few weeks ago [1] and
> got good feedback which we’ve incorporated already [2, 3, 4].
>
> There are a couple tasks left:
>
>- HDFS-12273 for the UI. This should be completed in the next couple
>days.
>- HDFS-12284 for adding security. We can move this for v2 if not
>completed.
>
> We have deployed this in production for 2.7 and we did a few tests with
> trunk a few months ago.
>
> This week, I’m rebasing to trunk (last one was a couple weeks ago) and test
> trunk in one of our test clusters.
>
>
> Finally, note that all the functionality is in the Router (a new component)
> so everything is isolated.
>
> In addition, no new APIs have been added and we rely fully in
> ClientProtocol.
>
>
>
> I’d like to thank the people at Microsoft (specially, Jason, Ricardo,
> Chris, Subru, Jakob, Carlo and Giovanni), Twitter (Ming and Gera), LinkedIn
> (Zhe, Erik and Konstantin), and Cloudera (Andrew and Manoj) for
> the discussion and the ideas.
>
> Special thanks to Chris Douglas for the thorough reviews!
>
>
>
> Regards,
> Inigo
>
>
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201708.mbox/%3CCAB1dGgogTu6kHtkkYeUycmNv-H3RupfPF4Cd7rpuFi6vHGdBLg%40mail.gmail.com%3E
>
> [2] https://issues.apache.org/jira/browse/HDFS-12381
>
> [3] https://issues.apache.org/jira/browse/HDFS-12430
>
> [4] https://issues.apache.org/jira/browse/HDFS-12450
>
-- 



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

2017-10-03 Thread Brahma Reddy Battula 
Thanks Andrew.

+1 (non binding)

--Built from source
--installed 3 node HA cluster
--Verified shell commands and UI
--Ran wordcount/pic jobs




On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang 
wrote:

> Hi all,
>
> Let me start, as always, by thanking the many, many contributors who helped
> with this release! I've prepared an RC0 for 3.0.0-beta1:
>
> http://home.apache.org/~wang/3.0.0-beta1-RC0/
>
> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
>
> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
> improvements, and feature enhancements. Notable additions include the
> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
> shaded client, and HDFS erasure coding pluggable policy support.
>
> I've done the traditional testing of running a Pi job on a pseudo cluster.
> My +1 to start.
>
> We're working internally on getting this run through our integration test
> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.
>
> Best,
> Andrew
>
-- 



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

2017-10-03 Thread Rushabh Shah 
Thank you Andrew for all your efforts.
+1 non-binding.
Built from source.
Started 1 node pseudo cluster.
Ran sleep and pi job.
Except for YARN-6999  I
didn't face any issues.


On Thu, Sep 28, 2017 at 7:04 PM, Andrew Wang 
wrote:

> Hi all,
>
> Let me start, as always, by thanking the many, many contributors who helped
> with this release! I've prepared an RC0 for 3.0.0-beta1:
>
> http://home.apache.org/~wang/3.0.0-beta1-RC0/
>
> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
>
> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
> improvements, and feature enhancements. Notable additions include the
> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
> shaded client, and HDFS erasure coding pluggable policy support.
>
> I've done the traditional testing of running a Pi job on a pseudo cluster.
> My +1 to start.
>
> We're working internally on getting this run through our integration test
> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.
>
> Best,
> Andrew
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-10-03 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/

[Oct 2, 2017 5:58:19 PM] (cliang) HADOOP-14915. method name is incorrect in 
ConfServlet. Contributed by
[Oct 2, 2017 6:26:53 PM] (cliang) HDFS-12552. Use slf4j instead of log4j in 
FSNamesystem. Contributed by
[Oct 2, 2017 8:03:32 PM] (Carlo Curino) YARN-3661. Basic Federation UI. 
(Contributed by Inigo Goiri via curino)
[Oct 2, 2017 8:11:47 PM] (aengineer) HADOOP-14752. 
TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky.
[Oct 2, 2017 8:18:00 PM] (cliang) HDFS-12271. Incorrect statement in Downgrade 
section of HDFS Rolling
[Oct 2, 2017 10:30:22 PM] (wangda) YARN-7259. Add size-based rolling policy to
[Oct 2, 2017 10:35:49 PM] (lei) HDFS-12569. Unset EC policy logs empty payload 
in edit log. (Lei (Eddy)
[Oct 2, 2017 10:40:00 PM] (aengineer) HDFS-12576. JournalNodes are getting 
started, even though
[Oct 3, 2017 1:14:44 AM] (subru) YARN-2037. Add work preserving restart support 
for Unmanaged AMs.
[Oct 3, 2017 2:04:49 AM] (sidharta) YARN-7226. Whitelisted variables do not 
support delayed variable
[Oct 3, 2017 3:14:54 AM] (aajisaka) MAPREDUCE-6971. Moving logging APIs over to 
slf4j in




-1 overall


The following subsystems voted -1:
unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.net.TestDNS 
   hadoop.crypto.key.kms.server.TestKMS 
   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.sls.TestReservationSystemInvariants 
   hadoop.yarn.sls.TestSLSRunner 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/whitespace-tabs.txt
  [1.2M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [152K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/patch-unit-hadoop-common-project_hadoop-kms.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [296K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/546/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [20K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

2017-10-03 Thread Marton, Elek 

+1 (non-binding)

* Built from the source
* Installed dockerized YARN/HADOOP cluster to a 20 node cluster 
(scheduled with nomad, configured from consul, docker host networking)
* Started example yarn jobs (teragen/terasort) and hdfs dfs commands + 
checking UIs


I noticed only two very minor issues (changelog of HADOOP-9902 didn't 
mention that I need a writable 'logs' dir, even with custom 
log4j.properties; and there was a space typo in yarn error message 
https://issues.apache.org/jira/browse/YARN-7279)


Marton

On 09/29/2017 02:04 AM, Andrew Wang wrote:

Hi all,

Let me start, as always, by thanking the many, many contributors who helped
with this release! I've prepared an RC0 for 3.0.0-beta1:

http://home.apache.org/~wang/3.0.0-beta1-RC0/

This vote will run five days, ending on Nov 3rd at 5PM Pacific.

beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
improvements, and feature enhancements. Notable additions include the
addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
shaded client, and HDFS erasure coding pluggable policy support.

I've done the traditional testing of running a Pi job on a pseudo cluster.
My +1 to start.

We're working internally on getting this run through our integration test
rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.

Best,
Andrew



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12401) Ozone: TestBlockDeletingService#testBlockDeletionTimeout sometimes timeout

2017-10-03 Thread Weiwei Yang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved HDFS-12401.

Resolution: Cannot Reproduce

> Ozone: TestBlockDeletingService#testBlockDeletionTimeout sometimes timeout
> --
>
> Key: HDFS-12401
> URL: https://issues.apache.org/jira/browse/HDFS-12401
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Weiwei Yang
>Priority: Minor
>
> {code}
> testBlockDeletionTimeout(org.apache.hadoop.ozone.container.common.TestBlockDeletingService)
>   Time elapsed: 100.383 sec  <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for condition. 
> Thread diagnostics:
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12583) Ozone: Fix swallow exceptions which makes hard to debug failures

2017-10-03 Thread Yiqun Lin (JIRA) 
Yiqun Lin created HDFS-12583:


 Summary: Ozone: Fix swallow exceptions which makes hard to debug 
failures
 Key: HDFS-12583
 URL: https://issues.apache.org/jira/browse/HDFS-12583
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Affects Versions: HDFS-7240
Reporter: Yiqun Lin
Assignee: Yiqun Lin


There are some places that swallow exceptions that makes client hard to debug 
the failure. For example, if we get xceiver client from xceiver client manager 
error, we only get the error info like this:
{noformat}
org.apache.hadoop.ozone.web.exceptions.OzoneException: Exception getting 
XceiverClient.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:119)
at 
com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createUsingDefault(StdValueInstantiator.java:243)
{noformat}
The error exception stack is missing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org