date:20170123

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

2017-01-23 Thread Allen Wittenauer


> On Jan 23, 2017, at 8:50 PM, Chris Douglas  wrote:
> 
> Thanks for all your work on this, Andrew. It's great to see the 3.x
> series moving forward.
> 
> If you were willing to modify the release notes and add the LICENSE to
> the jar, we don't need to reset the clock on the VOTE, IMO.

FWIW, I wrote a new version of the verify-license-files tool and attached it to 
HADOOP-13374.  This version actually verifies that the license and notice files 
in jars and wars matches the one in base of the (tarball) distribution.

ERROR: hadoop-client-api-3.0.0-alpha3-SNAPSHOT.jar: Missing a LICENSE file
ERROR: hadoop-client-api-3.0.0-alpha3-SNAPSHOT.jar: No valid NOTICE found

WARNING: hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar: Found 5 LICENSE 
files (0 were valid)
ERROR: hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar: No valid LICENSE 
found
WARNING: hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar: Found 3 NOTICE 
files (0 were valid)
ERROR: hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar: No valid NOTICE 
found

ERROR: hadoop-client-runtime-3.0.0-alpha3-SNAPSHOT.jar: No valid LICENSE found
ERROR: hadoop-client-runtime-3.0.0-alpha3-SNAPSHOT.jar: No valid NOTICE found

> What's the issue with the minicluster jar [1]? I tried to reproduce,
> but had no issues with 1.8.0_92-b14.

minicluster is kind of weird on filesystems that don't support mixed case, like 
OS X's default HFS+.

$  jar tf hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar | grep -i license
LICENSE.txt
license/
license/LICENSE
license/LICENSE.dom-documentation.txt
license/LICENSE.dom-software.txt
license/LICENSE.sax.txt
license/NOTICE
license/README.dom.txt
license/README.sax.txt
LICENSE
Grizzly_THIRDPARTYLICENSEREADME.txt



The problem here is that there is a 'license' directory and a file called 
'LICENSE'.  If this gets extracted by jar via jar xf, it will fail.  unzip can 
be made to extract it via an option like -o.  To make matters worse, none of 
these license files match the one in the generated tarball. :(



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

2017-01-23 Thread Chris Douglas

Thanks for all your work on this, Andrew. It's great to see the 3.x
series moving forward.

If you were willing to modify the release notes and add the LICENSE to
the jar, we don't need to reset the clock on the VOTE, IMO.

What's the issue with the minicluster jar [1]? I tried to reproduce,
but had no issues with 1.8.0_92-b14.

+1 Verified checksums, signature, built src tarball. -C

[1] 
hadoop-3.0.0-alpha2/share/hadoop/client/hadoop-client-minicluster-3.0.0-alpha2.jar


On Mon, Jan 23, 2017 at 10:51 AM, Andrew Wang  wrote:
> There are 5 JIRAs by my count that are missing release notes, which isn't
> that bad but could of course be improved. Four of those I had already
> checked earlier and forgot to follow up, and they were very minorly
> incompatible (affecting private APIs) or mistakenly marked incompatible.
>
> I'm not too concerned about the shaded minicluster since it's a new
> feature, this is an alpha, and we have an IT test against the shaded
> minicluster. Multiple files with the same name are actually also allowed by
> the zip standard, so it's not clear if there is a functionality bug.
>
> Could I get some additional PMC input on this vote? The most critical issue
> in my mind is the missing LICENSE on that one new artifact. If we end up
> spinning a new RC, I'll also handle the missing release notes that Allen
> mentioned.
>
> Thanks,
> Andrew
>
> On Sun, Jan 22, 2017 at 10:45 PM, Allen Wittenauer > wrote:
>
>>
>> > On Jan 22, 2017, at 9:05 PM, Allen Wittenauer 
>> wrote:
>> >
>> >
>> >
>> >
>> >
>> >> On Jan 20, 2017, at 2:36 PM, Andrew Wang 
>> wrote:
>> >>
>> >> http://home.apache.org/~wang/3.0.0-alpha2-RC0/
>> >
>> >   There are quite a few JIRA issues that need release notes.
>> >
>>
>>
>> One other thing, before I forget... I'm not sure the
>> hadoop-client-minicluster jar is getting built properly.  If you look
>> inside, you'll find a real mishmash of things, including files and
>> directories with the same names but different cases.  This means it won't
>> extract properly on OS X.  (jar xf on that jar file literally stack traces
>> on my El Capitan machine. Neat!)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

2017-01-23 Thread Andrew Wang

There are 5 JIRAs by my count that are missing release notes, which isn't
that bad but could of course be improved. Four of those I had already
checked earlier and forgot to follow up, and they were very minorly
incompatible (affecting private APIs) or mistakenly marked incompatible.

I'm not too concerned about the shaded minicluster since it's a new
feature, this is an alpha, and we have an IT test against the shaded
minicluster. Multiple files with the same name are actually also allowed by
the zip standard, so it's not clear if there is a functionality bug.

Could I get some additional PMC input on this vote? The most critical issue
in my mind is the missing LICENSE on that one new artifact. If we end up
spinning a new RC, I'll also handle the missing release notes that Allen
mentioned.

Thanks,
Andrew

On Sun, Jan 22, 2017 at 10:45 PM, Allen Wittenauer  wrote:

>
> > On Jan 22, 2017, at 9:05 PM, Allen Wittenauer 
> wrote:
> >
> >
> >
> >
> >
> >> On Jan 20, 2017, at 2:36 PM, Andrew Wang 
> wrote:
> >>
> >> http://home.apache.org/~wang/3.0.0-alpha2-RC0/
> >
> >   There are quite a few JIRA issues that need release notes.
> >
>
>
> One other thing, before I forget... I'm not sure the
> hadoop-client-minicluster jar is getting built properly.  If you look
> inside, you'll find a real mishmash of things, including files and
> directories with the same names but different cases.  This means it won't
> extract properly on OS X.  (jar xf on that jar file literally stack traces
> on my El Capitan machine. Neat!)

Re: [VOTE] Release cadence and EOL

2017-01-23 Thread Sangjin Lee

I'm going to stop the vote and go back to the discussion. It shouldn't be a
big surprise given the reservation we have so far.

I do hope there will be some actionable outcome as a result of that
discussion.

Regards,
Sangjin

On Mon, Jan 23, 2017 at 8:17 AM, Allen Wittenauer 
wrote:

>
> > On Jan 21, 2017, at 7:08 PM, Karthik Kambatla 
> wrote:
> >
> >   3. RM: some method to madness. Junping, for instance, is trying to roll
> >   a release with 2300 patches. It is a huge time investment. (Thanks
> again,
> >   Junping.) Smaller releases are easier to manage. A target release
> cadence,
> >   coupled with a process that encourages volunteering, IMO would lead to
> more
> >   committers doing releases.
>
>
> In the case of 2.8.0, that's on the original RM and "back port
> fever" that inflicts way too many committers.  2.8.0 has been sitting in a
> separate branch for over a year.  Of *course* it is going to be a
> disaster.  If the original RM had said "I don't have time, someone take
> over" after 3 months of it being left idle or another committer feeling as
> though they could take it or not everyone dumping everything they can in
> every release possible, it wouldn't be nearly as bad.
>
> Not only do we need to encourage volunteering, but we also need to
> encourage people to relinquish control. If the PMC wants to enact a
> cadence, then they also must be willing to step in when an RM is
> unresponsive and request someone else take over.  A message every three
> months saying "Yes, I'm still working on it." doesn't really help anyone,
> including the RM.
>
>
> > To conclude, the biggest value I see is us (the community) agreeing on
> good
> > practices for our releases and work towards that. Writing it down
> somewhere
> > makes it a little more formal like the compatibility stuff, even if it is
> > not enforceable.
>
> So it's exactly like the compatibility stuff. ;)
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release cadence and EOL

2017-01-23 Thread Allen Wittenauer

> On Jan 21, 2017, at 7:08 PM, Karthik Kambatla  wrote:
> 
>   3. RM: some method to madness. Junping, for instance, is trying to roll
>   a release with 2300 patches. It is a huge time investment. (Thanks again,
>   Junping.) Smaller releases are easier to manage. A target release cadence,
>   coupled with a process that encourages volunteering, IMO would lead to more
>   committers doing releases.

In the case of 2.8.0, that's on the original RM and "back port fever" 
that inflicts way too many committers.  2.8.0 has been sitting in a separate 
branch for over a year.  Of *course* it is going to be a disaster.  If the 
original RM had said "I don't have time, someone take over" after 3 months of 
it being left idle or another committer feeling as though they could take it or 
not everyone dumping everything they can in every release possible, it wouldn't 
be nearly as bad.

Not only do we need to encourage volunteering, but we also need to 
encourage people to relinquish control. If the PMC wants to enact a cadence, 
then they also must be willing to step in when an RM is unresponsive and 
request someone else take over.  A message every three months saying "Yes, I'm 
still working on it." doesn't really help anyone, including the RM.

> To conclude, the biggest value I see is us (the community) agreeing on good
> practices for our releases and work towards that. Writing it down somewhere
> makes it a little more formal like the compatibility stuff, even if it is
> not enforceable.

So it's exactly like the compatibility stuff. ;)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-01-23 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/

No changes




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.TestEncryptionZones 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 

Timed out junit tests :

   
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/diff-compile-javac-root.txt
  [160K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/diff-patch-shellcheck.txt
  [24K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [212K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/295/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11360) HDFS balancer need to appoint to a decided nameservice

2017-01-23 Thread Lantao Jin (JIRA)

Lantao Jin created HDFS-11360:
-

 Summary: HDFS balancer need to appoint to a decided nameservice
 Key: HDFS-11360
 URL: https://issues.apache.org/jira/browse/HDFS-11360
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer & mover
Reporter: Lantao Jin
Priority: Minor


With distcp configuration, there are two or more "nameservices" setting in 
hdfs-site.xml, for example:
{code}

dfs.nameservices
one-nn-ha,two--nn-ha

{code}

If the HDFS Balancer also launches in that node, it will create two IPC threads 
to connect both of NNs. And block removing also happens in the this Balancer. 
Although I didn't find any block removing between different nodes, the behavior 
still became weird and unexpected.
So the best way is adding a configuration that appoint to a decided nameservice 
to launch. I can offer a patch. Any better ideal? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11359) DFSAdmin report command support displaying maintenance state datanodes

2017-01-23 Thread Yiqun Lin (JIRA)

Yiqun Lin created HDFS-11359:


 Summary: DFSAdmin report command support displaying maintenance 
state datanodes
 Key: HDFS-11359
 URL: https://issues.apache.org/jira/browse/HDFS-11359
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: 3.0.0-alpha1
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor


The datanode's maintenance state info can be showed in webUI/JMX. But it can't 
be displayed via CLI. This JIRA will improve on this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

Re: [VOTE] Release cadence and EOL

Re: [VOTE] Release cadence and EOL

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[jira] [Created] (HDFS-11360) HDFS balancer need to appoint to a decided nameservice

[jira] [Created] (HDFS-11359) DFSAdmin report command support displaying maintenance state datanodes

8 matches

Site Navigation

Mail list logo

Footer information