[jira] [Work logged] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[ https://issues.apache.org/jira/browse/HIVE-21907?focusedWorklogId=439904&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439904 ] ASF GitHub Bot logged work on HIVE-21907: - Author: ASF GitHub Bot Created on: 02/Jun/20 06:09 Start Date: 02/Jun/20 06:09 Worklog Time Spent: 10m Work Description: pvary closed pull request #685: URL: https://github.com/apache/hive/pull/685 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439904) Time Spent: 1h 20m (was: 1h 10m) > Add a new LlapDaemon Management API method to set the daemon capacity > - > > Key: HIVE-21907 > URL: https://issues.apache.org/jira/browse/HIVE-21907 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21907.2.patch, HIVE-21907.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Add a new method to LlapManagementProtocol API which can disable an Llap node. > It would be even better, if we can dynamically set the number of executors > and the size of the wait queue. This way we can disable the node setting them > to 0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[
https://issues.apache.org/jira/browse/HIVE-21907?focusedWorklogId=266600&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266600
]
ASF GitHub Bot logged work on HIVE-21907:
-
Author: ASF GitHub Bot
Created on: 25/Jun/19 12:33
Start Date: 25/Jun/19 12:33
Worklog Time Spent: 10m
Work Description: pvary commented on pull request #685: HIVE-21907: Add a
new LlapDaemon Management API method to set the daemon capacity
URL: https://github.com/apache/hive/pull/685#discussion_r297162456
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
##
@@ -176,6 +181,31 @@ public TaskExecutorService(int numExecutors, int
waitQueueSize,
Futures.addCallback(future, new WaitQueueWorkerCallback());
}
+ /**
+ * Sets the TaskExecutorService capacity to the new values. Both the number
of executors and the
+ * queue size should be smaller than that original values, so we do not mess
up with the other
+ * settings. Setting smaller capacity will not cancel or reject already
executing or queued tasks
+ * in itself.
+ * @param newNumExecutors The new number of executors
+ * @param newWaitQueueSize The new number of wait queue size
+ */
+ @Override
+ public synchronized void setCapacity(int newNumExecutors, int
newWaitQueueSize) {
+if (newNumExecutors > configuredMaxExecutors) {
Review comment:
Added check and test
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 266600)
Time Spent: 1h 10m (was: 1h)
> Add a new LlapDaemon Management API method to set the daemon capacity
> -
>
> Key: HIVE-21907
> URL: https://issues.apache.org/jira/browse/HIVE-21907
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21907.2.patch, HIVE-21907.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Add a new method to LlapManagementProtocol API which can disable an Llap node.
> It would be even better, if we can dynamically set the number of executors
> and the size of the wait queue. This way we can disable the node setting them
> to 0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[ https://issues.apache.org/jira/browse/HIVE-21907?focusedWorklogId=266541&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266541 ] ASF GitHub Bot logged work on HIVE-21907: - Author: ASF GitHub Bot Created on: 25/Jun/19 11:06 Start Date: 25/Jun/19 11:06 Worklog Time Spent: 10m Work Description: pvary commented on pull request #685: HIVE-21907: Add a new LlapDaemon Management API method to set the daemon capacity URL: https://github.com/apache/hive/pull/685#discussion_r297131670 ## File path: llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java ## @@ -176,6 +181,31 @@ public TaskExecutorService(int numExecutors, int waitQueueSize, Futures.addCallback(future, new WaitQueueWorkerCallback()); } + /** + * Sets the TaskExecutorService capacity to the new values. Both the number of executors and the + * queue size should be smaller than that original values, so we do not mess up with the other Review comment: Container memory sizes are calculated based on the executors. Adding more executors and this way executing more containers can result in memory oversubscription. Added more specific comment This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266541) Time Spent: 1h (was: 50m) > Add a new LlapDaemon Management API method to set the daemon capacity > - > > Key: HIVE-21907 > URL: https://issues.apache.org/jira/browse/HIVE-21907 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21907.2.patch, HIVE-21907.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Add a new method to LlapManagementProtocol API which can disable an Llap node. > It would be even better, if we can dynamically set the number of executors > and the size of the wait queue. This way we can disable the node setting them > to 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[ https://issues.apache.org/jira/browse/HIVE-21907?focusedWorklogId=266473&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266473 ] ASF GitHub Bot logged work on HIVE-21907: - Author: ASF GitHub Bot Created on: 25/Jun/19 09:17 Start Date: 25/Jun/19 09:17 Worklog Time Spent: 10m Work Description: pvary commented on pull request #685: HIVE-21907: Add a new LlapDaemon Management API method to set the daemon capacity URL: https://github.com/apache/hive/pull/685#discussion_r297087947 ## File path: llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java ## @@ -40,6 +40,7 @@ import org.apache.hadoop.hive.llap.daemon.QueryFailedHandler; import org.apache.hadoop.hive.llap.daemon.SchedulerFragmentCompletingListener; import org.apache.hadoop.hive.llap.daemon.impl.LlapTokenChecker.LlapTokenInfo; +import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos; Review comment: Removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266473) Time Spent: 50m (was: 40m) > Add a new LlapDaemon Management API method to set the daemon capacity > - > > Key: HIVE-21907 > URL: https://issues.apache.org/jira/browse/HIVE-21907 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21907.2.patch, HIVE-21907.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Add a new method to LlapManagementProtocol API which can disable an Llap node. > It would be even better, if we can dynamically set the number of executors > and the size of the wait queue. This way we can disable the node setting them > to 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[ https://issues.apache.org/jira/browse/HIVE-21907?focusedWorklogId=265774&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265774 ] ASF GitHub Bot logged work on HIVE-21907: - Author: ASF GitHub Bot Created on: 24/Jun/19 15:56 Start Date: 24/Jun/19 15:56 Worklog Time Spent: 10m Work Description: odraese commented on pull request #685: HIVE-21907: Add a new LlapDaemon Management API method to set the daemon capacity URL: https://github.com/apache/hive/pull/685#discussion_r296790415 ## File path: llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java ## @@ -40,6 +40,7 @@ import org.apache.hadoop.hive.llap.daemon.QueryFailedHandler; import org.apache.hadoop.hive.llap.daemon.SchedulerFragmentCompletingListener; import org.apache.hadoop.hive.llap.daemon.impl.LlapTokenChecker.LlapTokenInfo; +import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos; Review comment: This seems to be an unused import. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 265774) Time Spent: 40m (was: 0.5h) > Add a new LlapDaemon Management API method to set the daemon capacity > - > > Key: HIVE-21907 > URL: https://issues.apache.org/jira/browse/HIVE-21907 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21907.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Add a new method to LlapManagementProtocol API which can disable an Llap node. > It would be even better, if we can dynamically set the number of executors > and the size of the wait queue. This way we can disable the node setting them > to 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[ https://issues.apache.org/jira/browse/HIVE-21907?focusedWorklogId=265772&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265772 ] ASF GitHub Bot logged work on HIVE-21907: - Author: ASF GitHub Bot Created on: 24/Jun/19 15:56 Start Date: 24/Jun/19 15:56 Worklog Time Spent: 10m Work Description: odraese commented on pull request #685: HIVE-21907: Add a new LlapDaemon Management API method to set the daemon capacity URL: https://github.com/apache/hive/pull/685#discussion_r296793593 ## File path: llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java ## @@ -176,6 +181,31 @@ public TaskExecutorService(int numExecutors, int waitQueueSize, Futures.addCallback(future, new WaitQueueWorkerCallback()); } + /** + * Sets the TaskExecutorService capacity to the new values. Both the number of executors and the + * queue size should be smaller than that original values, so we do not mess up with the other Review comment: What other settings? Maybe the comment could be more specific? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 265772) Time Spent: 20m (was: 10m) > Add a new LlapDaemon Management API method to set the daemon capacity > - > > Key: HIVE-21907 > URL: https://issues.apache.org/jira/browse/HIVE-21907 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21907.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Add a new method to LlapManagementProtocol API which can disable an Llap node. > It would be even better, if we can dynamically set the number of executors > and the size of the wait queue. This way we can disable the node setting them > to 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[
https://issues.apache.org/jira/browse/HIVE-21907?focusedWorklogId=265773&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265773
]
ASF GitHub Bot logged work on HIVE-21907:
-
Author: ASF GitHub Bot
Created on: 24/Jun/19 15:56
Start Date: 24/Jun/19 15:56
Worklog Time Spent: 10m
Work Description: odraese commented on pull request #685: HIVE-21907: Add
a new LlapDaemon Management API method to set the daemon capacity
URL: https://github.com/apache/hive/pull/685#discussion_r296793765
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
##
@@ -176,6 +181,31 @@ public TaskExecutorService(int numExecutors, int
waitQueueSize,
Futures.addCallback(future, new WaitQueueWorkerCallback());
}
+ /**
+ * Sets the TaskExecutorService capacity to the new values. Both the number
of executors and the
+ * queue size should be smaller than that original values, so we do not mess
up with the other
+ * settings. Setting smaller capacity will not cancel or reject already
executing or queued tasks
+ * in itself.
+ * @param newNumExecutors The new number of executors
+ * @param newWaitQueueSize The new number of wait queue size
+ */
+ @Override
+ public synchronized void setCapacity(int newNumExecutors, int
newWaitQueueSize) {
+if (newNumExecutors > configuredMaxExecutors) {
Review comment:
Should we also verify that the new values are >= zero?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 265773)
Time Spent: 0.5h (was: 20m)
> Add a new LlapDaemon Management API method to set the daemon capacity
> -
>
> Key: HIVE-21907
> URL: https://issues.apache.org/jira/browse/HIVE-21907
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21907.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Add a new method to LlapManagementProtocol API which can disable an Llap node.
> It would be even better, if we can dynamically set the number of executors
> and the size of the wait queue. This way we can disable the node setting them
> to 0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21907) Add a new LlapDaemon Management API method to set the daemon capacity
[ https://issues.apache.org/jira/browse/HIVE-21907?focusedWorklogId=265712&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265712 ] ASF GitHub Bot logged work on HIVE-21907: - Author: ASF GitHub Bot Created on: 24/Jun/19 13:37 Start Date: 24/Jun/19 13:37 Worklog Time Spent: 10m Work Description: pvary commented on pull request #685: HIVE-21907: Add a new LlapDaemon Management API method to set the daemon capacity URL: https://github.com/apache/hive/pull/685 3 main changes: - Test for TestEvictingPriorityBlockingQueue.java so we can be sure it handles queue size change - Test for TaskExecutorService.java so we can be sure that it will handle capacity changes correctly - API change for the new method and test for it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 265712) Time Spent: 10m Remaining Estimate: 0h > Add a new LlapDaemon Management API method to set the daemon capacity > - > > Key: HIVE-21907 > URL: https://issues.apache.org/jira/browse/HIVE-21907 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21907.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Add a new method to LlapManagementProtocol API which can disable an Llap node. > It would be even better, if we can dynamically set the number of executors > and the size of the wait queue. This way we can disable the node setting them > to 0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
