Re: [DISCUSS] Merge FairScheduler preemption overhaul (YARN-4752) to trunk
Forgot to mention - All the patches were reviewed before getting committed to the branch. - The changes are strictly internal to the scheduler. Most of it is limited to FairScheduler with minimal changes adding helper methods to common scheduler code. On Thu, Nov 10, 2016 at 4:46 PM, Karthik Kambatla wrote: > Hi folks > > We have been working on overhauling FairScheduler preemption on branch > YARN-4572. It is close to being ready for merge to trunk: > >1. Preemption considers individual ResourceRequests to satisfy. >(YARN-5605) >2. Preemption now works within a leaf queue and across sibling leaf >queues. (YARN-5605) >3. Comprehensive unit tests for app starvation and preemption - >minshare and fairshare. (YARN-5783 and YARN-5819, the latter is close to >commit) >4. TODO: Clean up the TODOs to remove my initials and replace them >with appropriate JIRAs. > > There are some unresolved sub-tasks in the umbrella JIRA, but none of them > are regressions in the new implementation. > > I just uploaded the cumulative patch to YARN-4752 for Jenkins verification > and will follow up on any issues that come up. > > Would like to hear your thoughts on the merge. > > Thanks > Karthik > > PS: Post facto, I feel a feature branch was unnecessary for this work. > Github PRs with multiple commits for ease of review would have been enough. >
[DISCUSS] Merge FairScheduler preemption overhaul (YARN-4752) to trunk
Hi folks We have been working on overhauling FairScheduler preemption on branch YARN-4572. It is close to being ready for merge to trunk: 1. Preemption considers individual ResourceRequests to satisfy. (YARN-5605) 2. Preemption now works within a leaf queue and across sibling leaf queues. (YARN-5605) 3. Comprehensive unit tests for app starvation and preemption - minshare and fairshare. (YARN-5783 and YARN-5819, the latter is close to commit) 4. TODO: Clean up the TODOs to remove my initials and replace them with appropriate JIRAs. There are some unresolved sub-tasks in the umbrella JIRA, but none of them are regressions in the new implementation. I just uploaded the cumulative patch to YARN-4752 for Jenkins verification and will follow up on any issues that come up. Would like to hear your thoughts on the merge. Thanks Karthik PS: Post facto, I feel a feature branch was unnecessary for this work. Github PRs with multiple commits for ease of review would have been enough.
Re: Jenkins patch-compile fails on trunk
This looks like bug caused by npm, looking into it now. Thanks, Wangda On Thu, Nov 10, 2016 at 4:00 PM, Zhe Zhang wrote: > https://builds.apache.org/job/PreCommit-HDFS-Build/17517/ > artifact/patchprocess/patch-compile-root.txt > > Looks related to yarn-ui? The HDFS-10872 patch doesn't touch YARN, but the > patch-compile failed at YARN. > > Thanks, > Zhe >
Jenkins patch-compile fails on trunk
https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/patch-compile-root.txt Looks related to yarn-ui? The HDFS-10872 patch doesn't touch YARN, but the patch-compile failed at YARN. Thanks, Zhe
Re: Updated 2.8.0-SNAPSHOT artifact
We could consider porting a limited number of critical fixes to branch-2.8. We'd need to be judicious about it so as not to prolong that cycle, however. Sangjin On Thu, Nov 10, 2016 at 6:16 AM, Eric Payne wrote: > How do we come to a resolution regarding whether or not re-cut branch-2.8 > or release it as it is (after fixing blockers)? > > There are some things in branch-2 that I would like to pull back into > branch-2.8, so a resolution to this question will affect how I proceed. > > Thanks, > -Eric > > -- > *From:* Karthik Kambatla > *To:* Ming Ma > *Cc:* Sangjin Lee ; Jason Lowe ; > Akira Ajisaka ; Brahma Reddy Battula < > brahmareddy.batt...@huawei.com>; Vinod Kumar Vavilapalli < > vino...@apache.org>; "common-...@hadoop.apache.org" < > common-...@hadoop.apache.org>; "hdfs-...@hadoop.apache.org" < > hdfs-...@hadoop.apache.org>; "mapreduce-...@hadoop.apache.org" < > mapreduce-...@hadoop.apache.org>; "yarn-dev@hadoop.apache.org" < > yarn-dev@hadoop.apache.org> > *Sent:* Thursday, November 10, 2016 1:56 AM > > *Subject:* Re: Updated 2.8.0-SNAPSHOT artifact > > If there is interest in releasing off of branch-2.8, we should definitely > do that. As Sangjin mentioned, there might be value in doing 2.9 off > branch-2 too. > > How do we go about maintenance releases along those minor lines, and when > would we discontinue 2.6.x/2.7.x releases? > > On Wed, Nov 9, 2016 at 12:06 PM, Ming Ma wrote: > > > I would also prefer releasing current 2.8 branch sooner. There are > several > > incomplete features in branch-2 such as YARN-914 and HDFS-7877 that are > > better served if we can complete them in the next major release. Letting > > them span across multiple releases might not be desirable as there could > be > > some potential compatibility issues involved. Therefore if we recut 2.8 > it > > means we have to work on those items before the new 2.8 is released which > > could cause major delay on the schedule. > > > > On Mon, Nov 7, 2016 at 10:37 AM, Sangjin Lee wrote: > > > >> +1. Resetting the 2.8 effort and the branch at this point may be > >> counter-productive. IMO we should focus on resolving the remaining > >> blockers > >> and getting it out the door. I also think that we should seriously > >> consider > >> 2.9 as well, as a fairly large number of changes have accumulated in > >> branch-2 (over branch-2.8). > >> > >> > >> Sangjin > >> > >> On Fri, Nov 4, 2016 at 3:38 PM, Jason Lowe > > >> wrote: > >> > >> > At this point my preference would be to do the most expeditious thing > to > >> > release 2.8, whether that's sticking with the branch-2.8 we have today > >> or > >> > re-cutting it on branch-2. Doing a quick JIRA query, there's been > >> almost > >> > 2,400 JIRAs resolved in 2.8.0 (1). For many of them, it's well-past > >> time > >> > they saw a release vehicle. If re-cutting the branch means we have to > >> wrap > >> > up a few extra things that are still in-progress on branch-2 or add a > >> few > >> > more blockers to the list before we release then I'd rather stay where > >> > we're at and ship it ASAP. > >> > > >> > Jason > >> > (1) https://issues.apache.org/jira/issues/?jql=project%20in% > >> > 20%28hadoop%2C%20yarn%2C%20mapreduce%2C%20hdfs%29% > >> > 20and%20resolution%20%3D%20Fixed%20and%20fixVersion%20%3D%202.8.0 > >> > > >> > > >> > > >> > > >> > > >> >On Tuesday, October 25, 2016 5:31 PM, Karthik Kambatla < > >> > ka...@cloudera.com> wrote: > >> > > >> > > >> > Is there value in releasing current branch-2.8? Aren't we better off > >> > re-cutting the branch off of branch-2? > >> > > >> > On Tue, Oct 25, 2016 at 12:20 AM, Akira Ajisaka < > >> > ajisa...@oss.nttdata.co.jp> > >> > wrote: > >> > > >> > > It's almost a year since branch-2.8 has cut. > >> > > I'm thinking we need to release 2.8.0 ASAP. > >> > > > >> > > According to the following list, there are 5 blocker and 6 critical > >> > issues. > >> > > https://issues.apache.org/jira/issues/?filter=12334985 > >> > > > >> > > Regards, > >> > > Akira > >> > > > >> > > > >> > > On 10/18/16 10:47, Brahma Reddy Battula wrote: > >> > > > >> > >> Hi Vinod, > >> > >> > >> > >> Any plan on first RC for branch-2.8 ? I think, it has been long > time. > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> --Brahma Reddy Battula > >> > >> > >> > >> -Original Message- > >> > >> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] > >> > >> Sent: 20 August 2016 00:56 > >> > >> To: Jonathan Eagles > >> > >> Cc: common-...@hadoop.apache.org > >> > >> Subject: Re: Updated 2.8.0-SNAPSHOT artifact > >> > >> > >> > >> Jon, > >> > >> > >> > >> That is around the time when I branched 2.8, so I guess you were > >> getting > >> > >> SNAPSHOT artifacts till then from the branch-2 nightly builds. > >> > >> > >> > >> If you need it, we can set up SNAPSHOT builds. Or just wait for the > >> > first > >> > >> RC, which is around the corner. > >> > >> > >> > >> +Vinod > >> > >> > >> > >> On Jul 28, 2016, at 4:27 PM, Jo
Re: Updated 2.8.0-SNAPSHOT artifact
On Wed, Nov 9, 2016 at 11:56 PM, Karthik Kambatla wrote: > If there is interest in releasing off of branch-2.8, we should definitely > do that. As Sangjin mentioned, there might be value in doing 2.9 off > branch-2 too. > > How do we go about maintenance releases along those minor lines, and when > would we discontinue 2.6.x/2.7.x releases? > Per proposal on the release cadence and EOL discussed in another thread, once 2.8 is released probably the 2.6.x line would be EOLed. And that doesn't sound unreasonable. > > On Wed, Nov 9, 2016 at 12:06 PM, Ming Ma wrote: > >> I would also prefer releasing current 2.8 branch sooner. There are >> several incomplete features in branch-2 such as YARN-914 and HDFS-7877 that >> are better served if we can complete them in the next major release. >> Letting them span across multiple releases might not be desirable as there >> could be some potential compatibility issues involved. Therefore if we >> recut 2.8 it means we have to work on those items before the new 2.8 is >> released which could cause major delay on the schedule. >> >> On Mon, Nov 7, 2016 at 10:37 AM, Sangjin Lee wrote: >> >>> +1. Resetting the 2.8 effort and the branch at this point may be >>> counter-productive. IMO we should focus on resolving the remaining >>> blockers >>> and getting it out the door. I also think that we should seriously >>> consider >>> 2.9 as well, as a fairly large number of changes have accumulated in >>> branch-2 (over branch-2.8). >>> >>> >>> Sangjin >>> >>> On Fri, Nov 4, 2016 at 3:38 PM, Jason Lowe >>> wrote: >>> >>> > At this point my preference would be to do the most expeditious thing >>> to >>> > release 2.8, whether that's sticking with the branch-2.8 we have today >>> or >>> > re-cutting it on branch-2. Doing a quick JIRA query, there's been >>> almost >>> > 2,400 JIRAs resolved in 2.8.0 (1). For many of them, it's well-past >>> time >>> > they saw a release vehicle. If re-cutting the branch means we have to >>> wrap >>> > up a few extra things that are still in-progress on branch-2 or add a >>> few >>> > more blockers to the list before we release then I'd rather stay where >>> > we're at and ship it ASAP. >>> > >>> > Jason >>> > (1) https://issues.apache.org/jira/issues/?jql=project%20in% >>> > 20%28hadoop%2C%20yarn%2C%20mapreduce%2C%20hdfs%29% >>> > 20and%20resolution%20%3D%20Fixed%20and%20fixVersion%20%3D%202.8.0 >>> > >>> > >>> > >>> > >>> > >>> > On Tuesday, October 25, 2016 5:31 PM, Karthik Kambatla < >>> > ka...@cloudera.com> wrote: >>> > >>> > >>> > Is there value in releasing current branch-2.8? Aren't we better off >>> > re-cutting the branch off of branch-2? >>> > >>> > On Tue, Oct 25, 2016 at 12:20 AM, Akira Ajisaka < >>> > ajisa...@oss.nttdata.co.jp> >>> > wrote: >>> > >>> > > It's almost a year since branch-2.8 has cut. >>> > > I'm thinking we need to release 2.8.0 ASAP. >>> > > >>> > > According to the following list, there are 5 blocker and 6 critical >>> > issues. >>> > > https://issues.apache.org/jira/issues/?filter=12334985 >>> > > >>> > > Regards, >>> > > Akira >>> > > >>> > > >>> > > On 10/18/16 10:47, Brahma Reddy Battula wrote: >>> > > >>> > >> Hi Vinod, >>> > >> >>> > >> Any plan on first RC for branch-2.8 ? I think, it has been long >>> time. >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> --Brahma Reddy Battula >>> > >> >>> > >> -Original Message- >>> > >> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] >>> > >> Sent: 20 August 2016 00:56 >>> > >> To: Jonathan Eagles >>> > >> Cc: common-...@hadoop.apache.org >>> > >> Subject: Re: Updated 2.8.0-SNAPSHOT artifact >>> > >> >>> > >> Jon, >>> > >> >>> > >> That is around the time when I branched 2.8, so I guess you were >>> getting >>> > >> SNAPSHOT artifacts till then from the branch-2 nightly builds. >>> > >> >>> > >> If you need it, we can set up SNAPSHOT builds. Or just wait for the >>> > first >>> > >> RC, which is around the corner. >>> > >> >>> > >> +Vinod >>> > >> >>> > >> On Jul 28, 2016, at 4:27 PM, Jonathan Eagles >>> wrote: >>> > >>> >>> > >>> Latest snapshot is uploaded in Nov 2015, but checkins are still >>> coming >>> > >>> in quite frequently. >>> > >>> https://repository.apache.org/content/repositories/snapshots >>> /org/apach >>> > >>> e/hadoop/hadoop-yarn-api/ >>> > >>> >>> > >>> Are there any plans to start producing updated SNAPSHOT artifacts >>> for >>> > >>> current hadoop development lines? >>> > >>> >>> > >> >>> > >> >>> > >> >>> - >>> > >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >>> > >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >>> > >> >>> > >> >>> > >> >>> - >>> > >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >>> > >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >>> > >> >>> > >> >>> > > >>> > > ---
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/151/ [Nov 9, 2016 1:02:40 PM] (aajisaka) HADOOP-13800. Remove unused HADOOP_AUDIT_LOGGER from hadoop-env.sh. [Nov 9, 2016 3:15:11 PM] (Arun Suresh) YARN-5833. Addendum patch to include missing changes to yarn-default.xml [Nov 9, 2016 5:07:12 PM] (xiao) HADOOP-13590. Retry until TGT expires even if the UGI renewal thread [Nov 9, 2016 5:16:50 PM] (weichiu) HDFS-11056. Concurrent append and read operations lead to checksum [Nov 9, 2016 5:33:00 PM] (xiao) HADOOP-13346. DelegationTokenAuthenticationHandler writes via closed [Nov 9, 2016 9:11:37 PM] (templedf) YARN-4329. [YARN-5437] Allow fetching exact reason as to why a submitted [Nov 9, 2016 11:30:05 PM] (naganarasimha_gr) YARN-4498. Application level node labels stats to be available in REST [Nov 10, 2016 12:08:05 AM] (jianhe) YARN-5611. Provide an API to update lifetime of an application. [Nov 10, 2016 12:12:30 AM] (naganarasimha_gr) YARN-5856. Unnecessary duplicate start container request sent to NM [Nov 10, 2016 1:16:07 AM] (xiao) HDFS-11120. TestEncryptionZones should waitActive. Contributed by John [Nov 10, 2016 6:11:34 AM] (varunsaxena) YARN-5862. TestDiskFailures.testLocalDirsFailures failed (Yufei Gu via [Nov 10, 2016 6:40:03 AM] (varunsaxena) YARN-5843. Incorrect documentation for timeline service [Nov 10, 2016 7:44:02 AM] (kasha) YARN-5453. FairScheduler#update may skip update demand resource of child [Nov 10, 2016 11:21:33 AM] (vinayakumarb) HDFS-9337. Validate required params for WebHDFS requests (Contributed by -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestEncryptedTransfer hadoop.hdfs.TestDistributedFileSystem hadoop.hdfs.TestLeaseRecoveryStriped hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers hadoop.mapred.pipes.TestPipeApplication Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.mapred.TestMRIntermediateDataEncryption compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/151/artifact/out/patch-compile-root.txt [172K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/151/artifact/out/patch-compile-root.txt [172K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/151/artifact/out/patch-compile-root.txt [172K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/151/artifact/out/
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/ [Nov 9, 2016 8:11:25 AM] (Arun Suresh) YARN-5823. Update NMTokens in case of requests with only opportunistic [Nov 9, 2016 12:34:40 PM] (rkanter) YARN-5736. YARN container executor config does not handle white space [Nov 9, 2016 1:02:40 PM] (aajisaka) HADOOP-13800. Remove unused HADOOP_AUDIT_LOGGER from hadoop-env.sh. [Nov 9, 2016 3:15:11 PM] (Arun Suresh) YARN-5833. Addendum patch to include missing changes to yarn-default.xml [Nov 9, 2016 5:07:12 PM] (xiao) HADOOP-13590. Retry until TGT expires even if the UGI renewal thread [Nov 9, 2016 5:16:50 PM] (weichiu) HDFS-11056. Concurrent append and read operations lead to checksum [Nov 9, 2016 5:33:00 PM] (xiao) HADOOP-13346. DelegationTokenAuthenticationHandler writes via closed [Nov 9, 2016 9:11:37 PM] (templedf) YARN-4329. [YARN-5437] Allow fetching exact reason as to why a submitted [Nov 9, 2016 11:30:05 PM] (naganarasimha_gr) YARN-4498. Application level node labels stats to be available in REST [Nov 10, 2016 12:08:05 AM] (jianhe) YARN-5611. Provide an API to update lifetime of an application. [Nov 10, 2016 12:12:30 AM] (naganarasimha_gr) YARN-5856. Unnecessary duplicate start container request sent to NM [Nov 10, 2016 1:16:07 AM] (xiao) HDFS-11120. TestEncryptionZones should waitActive. Contributed by John [Nov 10, 2016 6:11:34 AM] (varunsaxena) YARN-5862. TestDiskFailures.testLocalDirsFailures failed (Yufei Gu via [Nov 10, 2016 6:40:03 AM] (varunsaxena) YARN-5843. Incorrect documentation for timeline service -1 overall The following subsystems voted -1: asflicense findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.tools.TestDFSAdmin hadoop.mapred.pipes.TestPipeApplication hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/whitespace-tabs.txt [1.3M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [4.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [332K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [92K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [68K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [316K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/221/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.or
Re: Updated 2.8.0-SNAPSHOT artifact
How do we come to a resolution regarding whether or not re-cut branch-2.8 or release it as it is (after fixing blockers)? There are some things in branch-2 that I would like to pull back into branch-2.8, so a resolution to this question will affect how I proceed. Thanks,-Eric From: Karthik Kambatla To: Ming Ma Cc: Sangjin Lee ; Jason Lowe ; Akira Ajisaka ; Brahma Reddy Battula ; Vinod Kumar Vavilapalli ; "common-...@hadoop.apache.org" ; "hdfs-...@hadoop.apache.org" ; "mapreduce-...@hadoop.apache.org" ; "yarn-dev@hadoop.apache.org" Sent: Thursday, November 10, 2016 1:56 AM Subject: Re: Updated 2.8.0-SNAPSHOT artifact If there is interest in releasing off of branch-2.8, we should definitely do that. As Sangjin mentioned, there might be value in doing 2.9 off branch-2 too. How do we go about maintenance releases along those minor lines, and when would we discontinue 2.6.x/2.7.x releases? On Wed, Nov 9, 2016 at 12:06 PM, Ming Ma wrote: > I would also prefer releasing current 2.8 branch sooner. There are several > incomplete features in branch-2 such as YARN-914 and HDFS-7877 that are > better served if we can complete them in the next major release. Letting > them span across multiple releases might not be desirable as there could be > some potential compatibility issues involved. Therefore if we recut 2.8 it > means we have to work on those items before the new 2.8 is released which > could cause major delay on the schedule. > > On Mon, Nov 7, 2016 at 10:37 AM, Sangjin Lee wrote: > >> +1. Resetting the 2.8 effort and the branch at this point may be >> counter-productive. IMO we should focus on resolving the remaining >> blockers >> and getting it out the door. I also think that we should seriously >> consider >> 2.9 as well, as a fairly large number of changes have accumulated in >> branch-2 (over branch-2.8). >> >> >> Sangjin >> >> On Fri, Nov 4, 2016 at 3:38 PM, Jason Lowe >> wrote: >> >> > At this point my preference would be to do the most expeditious thing to >> > release 2.8, whether that's sticking with the branch-2.8 we have today >> or >> > re-cutting it on branch-2. Doing a quick JIRA query, there's been >> almost >> > 2,400 JIRAs resolved in 2.8.0 (1). For many of them, it's well-past >> time >> > they saw a release vehicle. If re-cutting the branch means we have to >> wrap >> > up a few extra things that are still in-progress on branch-2 or add a >> few >> > more blockers to the list before we release then I'd rather stay where >> > we're at and ship it ASAP. >> > >> > Jason >> > (1) https://issues.apache.org/jira/issues/?jql=project%20in% >> > 20%28hadoop%2C%20yarn%2C%20mapreduce%2C%20hdfs%29% >> > 20and%20resolution%20%3D%20Fixed%20and%20fixVersion%20%3D%202.8.0 >> > >> > >> > >> > >> > >> > On Tuesday, October 25, 2016 5:31 PM, Karthik Kambatla < >> > ka...@cloudera.com> wrote: >> > >> > >> > Is there value in releasing current branch-2.8? Aren't we better off >> > re-cutting the branch off of branch-2? >> > >> > On Tue, Oct 25, 2016 at 12:20 AM, Akira Ajisaka < >> > ajisa...@oss.nttdata.co.jp> >> > wrote: >> > >> > > It's almost a year since branch-2.8 has cut. >> > > I'm thinking we need to release 2.8.0 ASAP. >> > > >> > > According to the following list, there are 5 blocker and 6 critical >> > issues. >> > > https://issues.apache.org/jira/issues/?filter=12334985 >> > > >> > > Regards, >> > > Akira >> > > >> > > >> > > On 10/18/16 10:47, Brahma Reddy Battula wrote: >> > > >> > >> Hi Vinod, >> > >> >> > >> Any plan on first RC for branch-2.8 ? I think, it has been long time. >> > >> >> > >> >> > >> >> > >> >> > >> --Brahma Reddy Battula >> > >> >> > >> -Original Message- >> > >> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] >> > >> Sent: 20 August 2016 00:56 >> > >> To: Jonathan Eagles >> > >> Cc: common-...@hadoop.apache.org >> > >> Subject: Re: Updated 2.8.0-SNAPSHOT artifact >> > >> >> > >> Jon, >> > >> >> > >> That is around the time when I branched 2.8, so I guess you were >> getting >> > >> SNAPSHOT artifacts till then from the branch-2 nightly builds. >> > >> >> > >> If you need it, we can set up SNAPSHOT builds. Or just wait for the >> > first >> > >> RC, which is around the corner. >> > >> >> > >> +Vinod >> > >> >> > >> On Jul 28, 2016, at 4:27 PM, Jonathan Eagles >> wrote: >> > >>> >> > >>> Latest snapshot is uploaded in Nov 2015, but checkins are still >> coming >> > >>> in quite frequently. >> > >>> https://repository.apache.org/content/repositories/snapshots >> /org/apach >> > >>> e/hadoop/hadoop-yarn-api/ >> > >>> >> > >>> Are there any plans to start producing updated SNAPSHOT artifacts >> for >> > >>> current hadoop development lines? >> > >>> >> > >> >> > >> >> > >> >> - >> > >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> > >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> > >> >> > >> >> > >> --
[jira] [Created] (YARN-5867) DirectoryCollection#checkDirs can cause incorrect permission of nmlocal dir
Bibin A Chundatt created YARN-5867:
--
Summary: DirectoryCollection#checkDirs can cause incorrect
permission of nmlocal dir
Key: YARN-5867
URL: https://issues.apache.org/jira/browse/YARN-5867
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Steps to reproduce
===
# Set umask to 027 for user
# Start nodemanager with nmlocal dir configured
nmlocal dir permission is *755*
{{LocalDirsHandlerService#serviceInit}}
{code}
FsPermission perm = new FsPermission((short)0755);
boolean createSucceeded = localDirs.createNonExistentDirs(localFs, perm);
createSucceeded &= logDirs.createNonExistentDirs(localFs, perm);
{code}
# After startup delete the nmlocal dir and wait for {{MonitoringTimerTask}} to
run (simulation using delete)
# Now check the permission of {{nmlocal dir}} will be *750*
*Root Cause*
{{DirectoryCollection#testDirs}} checks as following
{code}
// create a random dir to make sure fs isn't in read-only mode
verifyDirUsingMkdir(testDir);
{code}
which cause a new Random directory to be create in {{localdir}} using
{{DiskChecker.checkDir(dir)}} -> {{!mkdirsWithExistsCheck(dir)}} causing the
nmlocal dir to be created with wrong permission. *750*
Few application fail to container launch due to permission denied.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
