Hi Eric, Thanks for helping in verifying the release.
Post YARN-7370, preemption configs are refreshable. I tried to test by making some changes in capacity-scheduler.xml and invoking yarn rmadmin -refreshQueues. I can see the changes reflected as per logs after refresh. Could you please help to give some more scenarios so that i can try to reproduce. Meanwhile i ll try some other combinations as well and let you know. .... reservationsContinueLooking = true *preemptionDisabled = true* defaultAppPriorityPerQueue = 0 priority = 0 maxLifetime = -1 seconds defaultLifetime = -1 seconds 2018-11-29 06:25:53,792 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=sunilgovindan IP=127.0.0.1 OPERATION=refreshQueues TARGET=AdminService RESULT=SUCCESS 2018-11-29 06:25:55,900 INFO org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy: Capacity Scheduler configuration changed, updated preemption properties to: max_ignored_over_capacity = 0.1 natural_termination_factor = 0.2 max_wait_before_kill = 15000 monitoring_interval = 3000 *total_preemption_per_round = 0.4* observe_only = false lazy-preemption-enabled = false *intra-queue-preemption.enabled = false* *intra-queue-preemption.max-allowable-limit = 0.4* intra-queue-preemption.minimum-threshold = 0.5 intra-queue-preemption.preemption-order-policy = USERLIMIT_FIRST priority-utilization.underutilized-preemption.enabled = false select_based_on_reserved_containers = false additional_res_balance_based_on_reserved_containers = false Preemption-to-balance-queue-enabled = false *now i disabled preemption for default queue and made some changed in intraqueue-preemption params.* reservationsContinueLooking = true *preemptionDisabled = false* defaultAppPriorityPerQueue = 0 priority = 0 maxLifetime = -1 seconds defaultLifetime = -1 seconds 2018-11-29 06:29:32,620 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=sunilgovindan IP=127.0.0.1 OPERATION=refreshQueues TARGET=AdminService RESULT=SUCCESS 2018-11-29 06:29:34,893 INFO org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy: Capacity Scheduler configuration changed, updated preemption properties to: max_ignored_over_capacity = 0.1 natural_termination_factor = 0.2 max_wait_before_kill = 15000 monitoring_interval = 3000 *total_preemption_per_round = 0.7* observe_only = false lazy-preemption-enabled = false *intra-queue-preemption.enabled = true* *intra-queue-preemption.max-allowable-limit = 0.5* intra-queue-preemption.minimum-threshold = 0.5 intra-queue-preemption.preemption-order-policy = USERLIMIT_FIRST priority-utilization.underutilized-preemption.enabled = false select_based_on_reserved_containers = false additional_res_balance_based_on_reserved_containers = false Preemption-to-balance-queue-enabled = false On Thu, Nov 29, 2018 at 4:19 AM Eric Payne <erichadoo...@yahoo.com.invalid> wrote: > Sunil, > > So, the basic symptoms are that if preemption is enabled on any queue, the > preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all > of the preemption-specific properties are set back to the default values. > > This was introduced in branch-3.1, so it is NOT new behavior for release > 3.2.0. I am still tracking down the cause. I will open a JIRA once I do > further investigation if there is not one already. > > This will be a problem for installations which use preemption and which > use the refreshQueues feature. > > Thanks, > -Eric > > > On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne < > eric.payne1...@yahoo.com> wrote: > > > > > > Sunil, thanks for all of the hard work on this release. > > I have discovered that queue refresh doesn't work in some cases. For > example, when I change > yarn.scheduler.capacity.root.default.disable_preemption, it doesn't take > effect unless I restart the RM. > > I am still investigating, but I thought I should bring this up asap. > > Thanks, > -Eric > > > > > On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G <sun...@apache.org> > wrote: > > > > > > Hi folks, > > > > Thanks to all contributors who helped in this release [1]. I have created > > first release candidate (RC0) for Apache Hadoop 3.2.0. > > > Artifacts for this RC are available here: > > http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/ > > > > RC tag in git is release-3.2.0-RC0. > > > > The maven artifacts are available via repository.apache.org at > > https://repository.apache.org/content/repositories/orgapachehadoop-1174/ > > > This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST. > > > > 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature > additions > > are the highlights of this release. > > 1. Node Attributes Support in YARN > > 2. Hadoop Submarine project for running Deep Learning workloads on YARN > > 3. Support service upgrade via YARN Service API and CLI > > 4. HDFS Storage Policy Satisfier > > 5. Support Windows Azure Storage - Blob file system in Hadoop > > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a > > 7. Improvements in Router-based HDFS federation > > > > Thanks to Wangda, Vinod, Marton for helping me in preparing the release. > > I have done few testing with my pseudo cluster. My +1 to start. > > > > Regards, > > Sunil > > > > [1] > > > https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0) > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved > ORDER BY fixVersion ASC > > --------------------------------------------------------------------- > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
