[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277186#comment-17277186 ]
Andras Gyori commented on YARN-10532: ------------------------------------- Thank you [~zhuqi] for the new patch. My additions are: * use the getLastSubmittedTimestamp instead of the explicit member variable because of the locking mechanism * AutoCreatedQueueDeletionEvent ** Please add license headerĀ ** The following line is redundant in AutoCreatedQueueDeletionPolicy: {code:java} assert null == scheduler : "Unexpected duplicate call to init"; {code} instanceof CapacityScheduler covers this case * ** monitoringInterval should only be a value of the QUEUE_AUTO_REMOVAL_MONITORING_INTERVAL, I am not quite sure about the other values there ** the getter methods should be marked visiblefortesting only ** there are unused setters/getters there ** reinitalize should be removed, I dont think we will use it anywhere ** getPolicyName should return the stringified name of the class, not a hardcoded value ** please rethink method names, for example initQueues makes no sense in this revision * CapacityScheduler ** the removeQueue could be rewritten to: {code:java} writeLock.lock(); try { LOG.info("Removing queue: " + queue.getQueuePath()); if (!(queue instanceof AbstractCSQueue)) { throw new SchedulerDynamicEditException( "The queue that we are asked " + "to remove (" + queue.getQueuePath() + ") is not a DynamicQueue"); } if (!((AbstractCSQueue) queue).isEligibleForAutoDeletion()) { LOG.warn("Queue " + queue.getQueuePath() + " is marked for deletion, but not eligible for deletion"); return; } ParentQueue parentQueue = (ParentQueue)queue.getParent(); if (parentQueue != null) { ((ParentQueue) queue.getParent()).removeChildQueue(queue); } else { throw new SchedulerDynamicEditException( "The queue " + queue.getQueuePath() + " can't removed normally."); } if (parentQueue.childQueues.contains(queue) || queueManager.getQueue(queue.getQueuePath()) != null) { throw new SchedulerDynamicEditException( "The queue " + queue.getQueuePath() + " has not been removed normally."); } LOG.info("Removed queue: " + queue.getQueuePath()); } finally { writeLock.unlock(); } {code} ** If a queue is not eligible for auto deletion, we should not throw an error, just a warning (this is a plausible scenario in case of a race condition) * queue-auto-removal-monitoring-interval is redundant with queue-expiration-time, but is a misleading name. Perhaps the queue-expiration-time is the better one, this should be preferred. * LeafQueue ** signalToSubmitToQueue should be renamed to signalSubmission ** I agree that the number of all applications should be taken into consideration in isEligibleForAutoDeletion * Please add more test cases and test the logic of the policy itself > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > -------------------------------------------------------------------------------------------- > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Wangda Tan > Assignee: Qi Zhu > Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch, YARN-10532.010.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org