Re: DUCC services statistics
Thank you Eddie and Jerry for the info. -Daniel Am 20.07.2018 um 00:34 schrieb Eddie Epstein: Hi, As you may see, the default DUCC pinger for UIMA-AS services scraps JMX stats from the service broker to report the number of producers and consumers, the queue depth and a few other things. This pinger also does stat reset operations on each call to the pinger, I think every 2 minutes. A custom pinger can easily be created that does not do the reset, or even the getMeta call if desired. The string that pingers return to DUCC are displayed as hover text over the entries in the "State" column, for example over "Available". Eddie On Thu, Jul 19, 2018 at 7:31 AM, Daniel Baumartz < bauma...@stud.uni-frankfurt.de> wrote: Hi, we have a DUCC installation with different services that are being used by jobs and external clients. We want to collect some statistics for a monitoring tool that are not included in the DUCC dashboard, e.g. how often and when a service/queue has been used, which services are used by which client at the moment... It looks like I could get these information by monitoring the ActiveMQ messages with an external program, or by using a custom Pinger? What would be the best way to handle this? Thanks, Daniel
DUCC services statistics
Hi, we have a DUCC installation with different services that are being used by jobs and external clients. We want to collect some statistics for a monitoring tool that are not included in the DUCC dashboard, e.g. how often and when a service/queue has been used, which services are used by which client at the moment... It looks like I could get these information by monitoring the ActiveMQ messages with an external program, or by using a custom Pinger? What would be the best way to handle this? Thanks, Daniel
UIMA AS deploy: get MessageTimeoutException object
Hi, I can see an MessageTimeoutException in the log when calling BaseUIMAAsynchronousEngine_impl.deploy(): org.apache.uima.aae.error.MessageTimeoutException: Delegate Service:ServiceName Has Timed Out While Processing CAS:null at org.apache.uima.aae.delegate.Delegate$2.run(Delegate.java:827) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) The timeout is actually expected because the service is not running, but I can only catch an exception like this, without the details: org.apache.uima.resource.ResourceInitializationException at org.apache.uima.adapter.jms.activemq.SpringContainerDeployer.waitForServiceNotification(SpringContainerDeployer.java:530) at org.apache.uima.adapter.jms.activemq.SpringContainerDeployer.initializeContainer(SpringContainerDeployer.java:314) at org.apache.uima.adapter.jms.activemq.SpringContainerDeployer.deploy(SpringContainerDeployer.java:434) at org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngine_impl.deploy(BaseUIMAAsynchronousEngine_impl.java:916) ... Caused by: org.apache.uima.resource.ResourceInitializationException at org.apache.uima.aae.controller.BaseAnalysisEngineController.stop(BaseAnalysisEngineController.java:2001) at org.apache.uima.aae.controller.BaseAnalysisEngineController.stop(BaseAnalysisEngineController.java:1898) at org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.stop(AggregateAnalysisEngineController_impl.java:3272) at org.apache.uima.aae.controller.BaseAnalysisEngineController.terminate(BaseAnalysisEngineController.java:2184) at org.apache.uima.aae.controller.BaseAnalysisEngineController.terminate(BaseAnalysisEngineController.java:2150) at org.apache.uima.aae.error.handler.GetMetaErrorHandler.handleError(GetMetaErrorHandler.java:104) at org.apache.uima.aae.error.ErrorHandlerChain.handle(ErrorHandlerChain.java:57) at org.apache.uima.aae.delegate.ControllerDelegate.handleError(ControllerDelegate.java:61) at org.apache.uima.aae.delegate.Delegate$2.run(Delegate.java:839) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) I tried to use the callback listeners, but they do not seem to get called, am I doing something wrong here? Or is there another way to get the MessageTimeoutException object during deploy? Thanks, Daniel
Re: DUCC Java API job info
Thanks for the json url. An addition to the DuccJobMonitor would be great! -Daniel On 19.04.2017 20:22, Lou DeGenaro wrote: There is no CLI query that gives the desired results. One could write a python script to visit http://uima-ducc-demo.apache.org:42133/ducc-servlet/json-format-aaData-jobs and parse the resulting json. Should there be a Jira to provide a CLI query capability equivalent of the Jobs page? Either a new query or the DuccJobMonitor could be enhanced. Lou. On Wed, Apr 19, 2017 at 10:26 AM, Daniel Baumartz < bauma...@stud.uni-frankfurt.de> wrote: Hi, I am trying to get the job info (start, duration and time until completion, basically the data from the web server jobs page) with the DUCC Java API. I was able to get some of the data (total, done...) using DuccJobMonitor, but I can't figure out how to get the others. Is there a way to access these? Thanks, Daniel
DUCC Java API job info
Hi, I am trying to get the job info (start, duration and time until completion, basically the data from the web server jobs page) with the DUCC Java API. I was able to get some of the data (total, done...) using DuccJobMonitor, but I can't figure out how to get the others. Is there a way to access these? Thanks, Daniel
Re: UIMA DUCC limit max memory of node
Hi Eddie, ok, I will try to explain with more detail, maybe this is not how ducc is being used normally. We want to set up some nodes which are not exclusively used for ducc. For example, one of the nodes may have 100 GB, but we want the usable memory for ducc to only be 50 GB, not all free memory. (We also want to limit the CPU usage, for example only use 32 of 64 cores, but we have not tried to set this up yet.) We could not find any setting to achieve this behavior, so we tried using cgroups to limit the max usable memory for ducc. This did not work because ducc gets its memory info from /proc/meminfo which ignores the cgroups settings. After reading through the code it seems only setting "ducc.agent.node.metrics.fake.memory.size" (not setting up test mode) is doing something similar to what we want: "Comment from NodeMemInfoCollector.java: if running ducc in simulation mode skip memory adjustment. Report free memory = fakeMemorySize". But I am not sure if we can use this safely since it is for testing. So we basically want to give ducc an upper limit of usable memory. I hope it is a bit more clear what we want to achieve. Thanks again, Daniel Zitat von Eddie Epstein <eaepst...@gmail.com>: Hi Daniel, For each node Ducc sums RSS for all "system" user processes and excludes that from Ducc usable memory on the node. System users are defined by a ducc.properties setting with default value: ducc.agent.node.metrics.sys.gid.max = 500 Ducc's simulation mode is intended for creating a scaled out cluster of fake nodes for testing purposes. The only mechanism available for reserving additional memory is to have Ducc run some dummy process that stays up forever. This could be a Ducc service that is automatically started when Ducc starts. This could get complicated for a heterogeneous set of machines and/or Ducc classes. Can you be more precise of what features you are looking for limiting resource use of Ducc machines? Thanks, Eddie On Mon, Oct 31, 2016 at 10:03 AM, Daniel Baumartz < bauma...@stud.uni-frankfurt.de> wrote: Hi, I am trying to set up nodes for Ducc that should not use all the memory on the machine. I tried to limit the memory with cgroups, but it seems Ducc is getting the memory info from /proc/meminfo which ignores the cgroups settings. Did I miss an option to specify the max usable memory? Could I safely use "ducc.agent.node.metrics.fake.memory.size" from the simulation settings? Or is there a better way to do this? Thanks, Daniel
UIMA DUCC limit max memory of node
Hi, I am trying to set up nodes for Ducc that should not use all the memory on the machine. I tried to limit the memory with cgroups, but it seems Ducc is getting the memory info from /proc/meminfo which ignores the cgroups settings. Did I miss an option to specify the max usable memory? Could I safely use "ducc.agent.node.metrics.fake.memory.size" from the simulation settings? Or is there a better way to do this? Thanks, Daniel
