[jira] [Created] (HADOOP-15958) Revisiting LICENSE and NOTICE files

[Akira Ajisaka (JIRA)]

Akira Ajisaka created HADOOP-15958:
--

 Summary: Revisiting LICENSE and NOTICE files
 Key: HADOOP-15958
 URL: https://issues.apache.org/jira/browse/HADOOP-15958
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Akira Ajisaka


Originally reported from [~jmclean]:
* NOTICE file incorrectly lists copyrights that shouldn't be there and mentions 
licenses such as MIT, BSD, and public domain that should be mentioned in 
LICENSE only.
* It's better to have a separate LICENSE and NOTICE for the source and binary 
releases.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Wilfred Spiegelenburg]

+1 (non binding)

- build from source on MacOSX 10.14.1, 1.8.0u181
- successful native build on Ubuntu 16.04.3
- confirmed the checksum and signature
- deployed a single node cluster  (jdk 1.8u191 / centos 7.5)

Wilfred

> On 23 Nov 2018, at 23:06, Sunil G  wrote:
> 
> Hi folks,
> 
> 
> 
> Thanks to all contributors who helped in this release [1]. I have created
> 
> first release candidate (RC0) for Apache Hadoop 3.2.0.
> 
> 
> Artifacts for this RC are available here:
> 
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
> 
> 
> 
> RC tag in git is release-3.2.0-RC0.
> 
> 
> 
> The maven artifacts are available via repository.apache.org at
> 
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
> 
> 
> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
> 
> 
> 
> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
> 
> are the highlights of this release.
> 
> 1. Node Attributes Support in YARN
> 
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
> 
> 3. Support service upgrade via YARN Service API and CLI
> 
> 4. HDFS Storage Policy Satisfier
> 
> 5. Support Windows Azure Storage - Blob file system in Hadoop
> 
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> 
> 7. Improvements in Router-based HDFS federation
> 
> 
> 
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
> 
> I have done few testing with my pseudo cluster. My +1 to start.
> 
> 
> 
> Regards,
> 
> Sunil
> 
> 
> 
> [1]
> 
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> 
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC


Wilfred Spiegelenburg | Software Engineer
cloudera.com 

[jira] [Created] (HADOOP-15957) WASB: Add asterisk wildcard support for Page Blob Dir Set

[Da Zhou (JIRA)]

Da Zhou created HADOOP-15957:


 Summary: WASB: Add asterisk wildcard support for Page Blob Dir Set
 Key: HADOOP-15957
 URL: https://issues.apache.org/jira/browse/HADOOP-15957
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/azure
Affects Versions: 3.2.0
Reporter: Da Zhou
Assignee: Da Zhou


In WASB, property "*fs.azure.page.blob.dir*" only support literal directory 
name.
We need to add support for wildcard '*' to represent for any directory name.
For example, the following pattern should be supported:

{code:java}
/dir1/dir2 
/dir1/*/dir3
/dir1/*/*/dir4
/dir1/*/*/file
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Sunil G]

Hi Eric,

Thanks for helping in verifying the release.

Post YARN-7370, preemption configs are refreshable. I tried to test by
making some changes in capacity-scheduler.xml and invoking yarn rmadmin
-refreshQueues.
I can see the changes reflected as per logs after refresh. Could you please
help to give some more scenarios so that i can try to reproduce.
Meanwhile i ll try some other combinations as well and let you know.


reservationsContinueLooking = true
*preemptionDisabled = true*
defaultAppPriorityPerQueue = 0
priority = 0
maxLifetime = -1 seconds
defaultLifetime = -1 seconds
2018-11-29 06:25:53,792 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
USER=sunilgovindan IP=127.0.0.1 OPERATION=refreshQueues TARGET=AdminService
RESULT=SUCCESS
2018-11-29 06:25:55,900 INFO
org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy:
Capacity Scheduler configuration changed, updated preemption properties to:
max_ignored_over_capacity = 0.1
natural_termination_factor = 0.2
max_wait_before_kill = 15000
monitoring_interval = 3000
*total_preemption_per_round = 0.4*
observe_only = false
lazy-preemption-enabled = false
*intra-queue-preemption.enabled = false*
*intra-queue-preemption.max-allowable-limit = 0.4*
intra-queue-preemption.minimum-threshold = 0.5
intra-queue-preemption.preemption-order-policy = USERLIMIT_FIRST
priority-utilization.underutilized-preemption.enabled = false
select_based_on_reserved_containers = false
additional_res_balance_based_on_reserved_containers = false
Preemption-to-balance-queue-enabled = false

*now i disabled preemption for default queue and made some changed in
intraqueue-preemption params.*

reservationsContinueLooking = true
*preemptionDisabled = false*
defaultAppPriorityPerQueue = 0
priority = 0
maxLifetime = -1 seconds
defaultLifetime = -1 seconds
2018-11-29 06:29:32,620 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
USER=sunilgovindan IP=127.0.0.1 OPERATION=refreshQueues TARGET=AdminService
RESULT=SUCCESS
2018-11-29 06:29:34,893 INFO
org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy:
Capacity Scheduler configuration changed, updated preemption properties to:
max_ignored_over_capacity = 0.1
natural_termination_factor = 0.2
max_wait_before_kill = 15000
monitoring_interval = 3000
*total_preemption_per_round = 0.7*
observe_only = false
lazy-preemption-enabled = false
*intra-queue-preemption.enabled = true*
*intra-queue-preemption.max-allowable-limit = 0.5*
intra-queue-preemption.minimum-threshold = 0.5
intra-queue-preemption.preemption-order-policy = USERLIMIT_FIRST
priority-utilization.underutilized-preemption.enabled = false
select_based_on_reserved_containers = false
additional_res_balance_based_on_reserved_containers = false
Preemption-to-balance-queue-enabled = false

On Thu, Nov 29, 2018 at 4:19 AM Eric Payne 
wrote:

> Sunil,
>
> So, the basic symptoms are that if preemption is enabled on any queue, the
> preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all
> of the preemption-specific properties are set back to the default values.
>
> This was introduced in branch-3.1, so it is NOT new behavior for release
> 3.2.0. I am still tracking down the cause. I will open a JIRA once I do
> further investigation if there is not one already.
>
> This will be a problem for installations which use preemption and which
> use the refreshQueues feature.
>
> Thanks,
> -Eric
>
>
> On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne <
> eric.payne1...@yahoo.com> wrote:
>
>
>
>
>
> Sunil, thanks for all of the hard work on this release.
>
> I have discovered that queue refresh doesn't work in some cases. For
> example, when I change
> yarn.scheduler.capacity.root.default.disable_preemption, it doesn't take
> effect unless I restart the RM.
>
> I am still investigating, but I thought I should bring this up asap.
>
> Thanks,
> -Eric
>
>
>
>
> On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G 
> wrote:
>
>
>
>
>
> Hi folks,
>
>
>
> Thanks to all contributors who helped in this release [1]. I have created
>
> first release candidate (RC0) for Apache Hadoop 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
>
>
>
> RC tag in git is release-3.2.0-RC0.
>
>
>
> The maven artifacts are available via repository.apache.org at
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
>
>
> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6. P

Re: Apache Hadoop 3.1.2 release plan

[Wangda Tan]

Hi Steve,

Is there any other JIRAs need to be released 3.1.2? I want to get 3.1.2
release starts earlier if possible.

Thanks,
Wangda

On Wed, Nov 21, 2018 at 5:31 PM Wangda Tan  wrote:

> Hi Steve,
>
> It gonna be really good if you could get the patch committed during US
> Thanksgiving holiday so I can start releasing starting next week.
>
> Thanks,
> Wangda
>
> On Mon, Nov 19, 2018 at 10:21 AM Wangda Tan  wrote:
>
>> Hi Steve,
>>
>> It's helpful if you could share any timeline about when this will be done?
>>
>> Thanks,
>> Wangda
>>
>> On Mon, Nov 19, 2018 at 6:46 AM Steve Loughran 
>> wrote:
>>
>>> I think I need to a quick rollback of something in S3A, for 3.0+; turns
>>> out that copyFromLocalFile() is also used to upload directories, and the
>>> high-performance one doesn't.
>>>
>>> https://issues.apache.org/jira/browse/HADOOP-15932
>>>
>>> The patch doesn't delete the new code, just takes it out of the
>>> execution path
>>>
>>>
>>>
>>> > On 16 Nov 2018, at 18:14, Wangda Tan  wrote:
>>> >
>>> > Just cleaned up all tickets and updated tickets which have target
>>> version
>>> > == 3.1.2 to 3.1.3.
>>> >
>>> > I will roll an RC next Monday, if you have any tickets need to be
>>> released
>>> > in 3.1.2, please let me know.
>>> >
>>> > Thanks,
>>> > Wangda
>>> >
>>> > On Wed, Oct 24, 2018 at 7:30 PM Vinod Kumar Vavilapalli <
>>> vino...@apache.org>
>>> > wrote:
>>> >
>>> >> 231 fixed JIRAs is already quite a bunch!
>>> >>
>>> >> I only see 7 JIRAs marked with Affects Version 3.1.2 and only one of
>>> them
>>> >> as blocker.
>>> >>
>>> >> Why not just release now as soon as there are no blockers?
>>> >>
>>> >> Thanks
>>> >> +Vinod
>>> >>
>>> >>> On Oct 24, 2018, at 4:36 PM, Wangda Tan  wrote:
>>> >>>
>>> >>> Hi, All
>>> >>>
>>> >>> We have released Apache Hadoop 3.1.1 on Aug 8, 2018. To further
>>> >>> improve the quality of the release, I plan to release 3.1.2
>>> >>> by Nov. The focus of 3.1.2 will be fixing blockers / critical bugs
>>> >>> and other enhancements. So far there are 231 JIRAs [1] have fix
>>> >>> version marked to 3.1.2
>>> >>>
>>> >>> I plan to cut branch-3.1 on Nov 15 and vote for RC on the same day.
>>> >>>
>>> >>> Please feel free to share your insights.
>>> >>>
>>> >>> Thanks,
>>> >>> Wangda Tan
>>> >>>
>>> >>> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
>>> >> Map/Reduce")
>>> >>> AND fixVersion = 3.1.2
>>> >>
>>> >>
>>>
>>>

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Eric Payne]

Sunil,

So, the basic symptoms are that if preemption is enabled on any queue, the 
preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all of 
the preemption-specific properties are set back to the default values.

This was introduced in branch-3.1, so it is NOT new behavior for release 3.2.0. 
I am still tracking down the cause. I will open a JIRA once I do further 
investigation if there is not one already.

This will be a problem for installations which use preemption and which use the 
refreshQueues feature.

Thanks,
-Eric


On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne 
 wrote: 





Sunil, thanks for all of the hard work on this release.

I have discovered that queue refresh doesn't work in some cases. For example, 
when I change yarn.scheduler.capacity.root.default.disable_preemption, it 
doesn't take effect unless I restart the RM.

I am still investigating, but I thought I should bring this up asap.

Thanks,
-Eric




On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G  
wrote: 





Hi folks,



Thanks to all contributors who helped in this release [1]. I have created

first release candidate (RC0) for Apache Hadoop 3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/



RC tag in git is release-3.2.0-RC0.



The maven artifacts are available via repository.apache.org at

https://repository.apache.org/content/repositories/orgapachehadoop-1174/


This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.



3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Chandni Singh]

+1 (non-binding)

- Build from source
- Launched example sleeper service
- Tested upgrade of sleeper service
- Tested cancellation of upgrade
- Tested express upgrade

On Wed, Nov 28, 2018 at 8:22 AM Peter Bacsko 
wrote:

> +1 (non-binding)
>
> - Built from source at tag 3.2.0-rc0 (Ubuntu 18.10, JDK1.8.0_191)
> - Verified checksums of hadoop-3.2.0.tar.gz
> - Installed on a 3-node physical cluster
> - Ran teragen/terasort/teravalidate
> - Ran distributed shell a couple of times
> - Checked UIs (RM, NM, DN, JHS)
>
> Peter
>
> On Wed, Nov 28, 2018 at 5:17 PM Jason Lowe  wrote:
>
> > Thanks for driving this release, Sunil!
> >
> > +1 (binding)
> >
> > - Verified signatures and digests
> > - Successfully performed a native build
> > - Deployed a single-node cluster
> > - Ran some sample jobs
> >
> > Jason
> >
> > On Fri, Nov 23, 2018 at 6:07 AM Sunil G  wrote:
> >
> > > Hi folks,
> > >
> > >
> > >
> > > Thanks to all contributors who helped in this release [1]. I have
> created
> > >
> > > first release candidate (RC0) for Apache Hadoop 3.2.0.
> > >
> > >
> > > Artifacts for this RC are available here:
> > >
> > > http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
> > >
> > >
> > >
> > > RC tag in git is release-3.2.0-RC0.
> > >
> > >
> > >
> > > The maven artifacts are available via repository.apache.org at
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
> > >
> > >
> > > This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm
> PST.
> > >
> > >
> > >
> > > 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> > > additions
> > >
> > > are the highlights of this release.
> > >
> > > 1. Node Attributes Support in YARN
> > >
> > > 2. Hadoop Submarine project for running Deep Learning workloads on YARN
> > >
> > > 3. Support service upgrade via YARN Service API and CLI
> > >
> > > 4. HDFS Storage Policy Satisfier
> > >
> > > 5. Support Windows Azure Storage - Blob file system in Hadoop
> > >
> > > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> > >
> > > 7. Improvements in Router-based HDFS federation
> > >
> > >
> > >
> > > Thanks to Wangda, Vinod, Marton for helping me in preparing the
> release.
> > >
> > > I have done few testing with my pseudo cluster. My +1 to start.
> > >
> > >
> > >
> > > Regards,
> > >
> > > Sunil
> > >
> > >
> > >
> > > [1]
> > >
> > >
> > >
> >
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> > >
> > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.2.0)
> > > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> > > ORDER BY fixVersion ASC
> > >
> >
>

[jira] [Created] (HADOOP-15956) Use relative resource URLs across WebUI components

[Greg Phillips (JIRA)]

Greg Phillips created HADOOP-15956:
--

 Summary: Use relative resource URLs across WebUI components
 Key: HADOOP-15956
 URL: https://issues.apache.org/jira/browse/HADOOP-15956
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Greg Phillips


Similar to HDFS-12961 there are absolute paths used for static resources in the 
WebUI for HDFS & KMS which can cause issues when attempting to access these 
pages via a reverse proxy. Using relative paths in all WebUI components will 
allow pages to render properly when using a reverse proxy.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Eric Payne]

Sunil, thanks for all of the hard work on this release.

I have discovered that queue refresh doesn't work in some cases. For example, 
when I change yarn.scheduler.capacity.root.default.disable_preemption, it 
doesn't take effect unless I restart the RM.

I am still investigating, but I thought I should bring this up asap.

Thanks,
-Eric




On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G  
wrote: 





Hi folks,



Thanks to all contributors who helped in this release [1]. I have created

first release candidate (RC0) for Apache Hadoop 3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/



RC tag in git is release-3.2.0-RC0.



The maven artifacts are available via repository.apache.org at

https://repository.apache.org/content/repositories/orgapachehadoop-1174/


This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.



3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Peter Bacsko]

+1 (non-binding)

- Built from source at tag 3.2.0-rc0 (Ubuntu 18.10, JDK1.8.0_191)
- Verified checksums of hadoop-3.2.0.tar.gz
- Installed on a 3-node physical cluster
- Ran teragen/terasort/teravalidate
- Ran distributed shell a couple of times
- Checked UIs (RM, NM, DN, JHS)

Peter

On Wed, Nov 28, 2018 at 5:17 PM Jason Lowe  wrote:

> Thanks for driving this release, Sunil!
>
> +1 (binding)
>
> - Verified signatures and digests
> - Successfully performed a native build
> - Deployed a single-node cluster
> - Ran some sample jobs
>
> Jason
>
> On Fri, Nov 23, 2018 at 6:07 AM Sunil G  wrote:
>
> > Hi folks,
> >
> >
> >
> > Thanks to all contributors who helped in this release [1]. I have created
> >
> > first release candidate (RC0) for Apache Hadoop 3.2.0.
> >
> >
> > Artifacts for this RC are available here:
> >
> > http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
> >
> >
> >
> > RC tag in git is release-3.2.0-RC0.
> >
> >
> >
> > The maven artifacts are available via repository.apache.org at
> >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1174/
> >
> >
> > This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
> >
> >
> >
> > 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> > additions
> >
> > are the highlights of this release.
> >
> > 1. Node Attributes Support in YARN
> >
> > 2. Hadoop Submarine project for running Deep Learning workloads on YARN
> >
> > 3. Support service upgrade via YARN Service API and CLI
> >
> > 4. HDFS Storage Policy Satisfier
> >
> > 5. Support Windows Azure Storage - Blob file system in Hadoop
> >
> > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> >
> > 7. Improvements in Router-based HDFS federation
> >
> >
> >
> > Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
> >
> > I have done few testing with my pseudo cluster. My +1 to start.
> >
> >
> >
> > Regards,
> >
> > Sunil
> >
> >
> >
> > [1]
> >
> >
> >
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> > ORDER BY fixVersion ASC
> >
>

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Jason Lowe]

Thanks for driving this release, Sunil!

+1 (binding)

- Verified signatures and digests
- Successfully performed a native build
- Deployed a single-node cluster
- Ran some sample jobs

Jason

On Fri, Nov 23, 2018 at 6:07 AM Sunil G  wrote:

> Hi folks,
>
>
>
> Thanks to all contributors who helped in this release [1]. I have created
>
> first release candidate (RC0) for Apache Hadoop 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
>
>
>
> RC tag in git is release-3.2.0-RC0.
>
>
>
> The maven artifacts are available via repository.apache.org at
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
>
>
> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>
> 7. Improvements in Router-based HDFS federation
>
>
>
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>
> I have done few testing with my pseudo cluster. My +1 to start.
>
>
>
> Regards,
>
> Sunil
>
>
>
> [1]
>
>
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC
>

[jira] [Created] (HADOOP-15955) declare that fs.s3a.ext. is a prefix for arbitrary extensions

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15955:
---

 Summary: declare that fs.s3a.ext. is a prefix for arbitrary 
extensions
 Key: HADOOP-15955
 URL: https://issues.apache.org/jira/browse/HADOOP-15955
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 3.2.0
Reporter: Steve Loughran


it turns out to be useful to use per-bucket config extensions for things like 
delegation tokens. 
To avoid third-party extensions to pollute the fs.s3a. namespace with their own 
config options (So meaning its inevitable things break when we add some new fs 
option), I propose the following policy

# applications/extensions MUST NOT add their own config options under fs.s3a.
# anyone is free to use "fs.s3a.ext.$product." options in s3a. 
# we promise not to go near those options in the S3A module
# it is left to people who add their own settings here to avoid clashing with 
others, This is why you should use a unique product name

With this specified (where?) then if someone uses some fs.s3a. option which 
gets broken by an updated s3a connector, well, we say "wontfix", —and indeed, 
get to tell them off for doing this.

Propose: similar for the other stores




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/

[Nov 27, 2018 7:31:22 AM] (nanda) HDDS-868. Handle quasi closed container 
replicas in SCM. Contributed by
[Nov 27, 2018 9:03:38 AM] (sunilg) YARN-9058. [CSI] YARN service fail to launch 
due to CSI changes.
[Nov 27, 2018 11:01:11 AM] (aajisaka) MAPREDUCE-7162. TestEvents#testEvents 
fails. Contributed by Zhaohui Xin.
[Nov 27, 2018 11:32:54 AM] (kube-backup) HDDS-766. Ozone shell create volume 
fails if volume name does not have a
[Nov 27, 2018 6:13:07 PM] (stevel) HADOOP-15940. ABFS: For HNS account, avoid 
unnecessary get call when
[Nov 27, 2018 8:39:54 PM] (stevel) HADOOP-15932. Oozie unable to create 
sharelib in s3a filesystem.
[Nov 27, 2018 11:36:59 PM] (eyang) YARN-9056.  Improved YARN service upgrade 
state logic for readiness




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.client.api.impl.TestAMRMClient 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/diff-compile-javac-root.txt
  [336K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/diff-patch-pylint.txt
  [40K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/diff-patch-shellcheck.txt
  [68K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/whitespace-eol.txt
  [9.3M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/whitespace-tabs.txt
  [1.1M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-ozone_common.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/branch-findbugs-hadoop-ozone_tools.txt
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/diff-javadoc-javadoc-root.txt
  [752K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [84K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/971/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [16K]