Re: DUCC org.apache.uima.util.InvalidXMLException and no logs
2014-11-27 11:44 GMT-06:00 Eddie Epstein eaepst...@gmail.com: Those are the only two log files? Should be a ducc.log (probably with no more info than on the console), and either one or both of the job driver logfiles: jd.out.log and jobid-JD-jdnode-jdpid.log. If for some reason the job driver failed to start, check the job driver agent log (the agent managing the System/JobDriver reservation) for more info on what happened. The job driver logs do not exist. I rebooted the machine and now it works. I'll take a look at the agent log next time.
Ducc: Rename failed
When running DUCC in cluster mode, I get Rename failed. The file
mentioned in the error message exists in the txt.processed/ directory.
The mount is via nfs (rw,sync,insecure).
org.apache.uima.resource.ResourceProcessException: Received Exception
In Message From Service on Queue:ducc.jd.queue.75 Broker:
tcp://10.0.0.164:61617?jms.useCompression=true Cas
Identifier:18acd63:149f6f562d3:-7fa6 Exception:{3}
at
org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl.sendAndReceiveCAS(BaseUIMAAsynchronousEngineCommon_impl.java:2230)
at
org.apache.uima.adapter.jms.client.BaseUIMAAsynchronousEngineCommon_impl.sendAndReceiveCAS(BaseUIMAAsynchronousEngineCommon_impl.java:2049)
at org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.java:145)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.uima.aae.error.UimaEEServiceException:
org.apache.uima.analysis_engine.AnalysisEngineProcessException
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendReply(JmsOutputChannel.java:932)
at
org.apache.uima.aae.controller.BaseAnalysisEngineController.handleAction(BaseAnalysisEngineController.java:1172)
at
org.apache.uima.aae.controller.PrimitiveAnalysisEngineController_impl.takeAction(PrimitiveAnalysisEngineController_impl.java:1145)
at
org.apache.uima.aae.error.handler.ProcessCasErrorHandler.handleError(ProcessCasErrorHandler.java:405)
at
org.apache.uima.aae.error.ErrorHandlerChain.handle(ErrorHandlerChain.java:57)
at
org.apache.uima.aae.controller.PrimitiveAnalysisEngineController_impl.process(PrimitiveAnalysisEngineController_impl.java:1065)
at
org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:121)
at
org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handleProcessRequestFromRemoteClient(ProcessRequestHandler_impl.java:543)
at
org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:1050)
at
org.apache.uima.aae.handler.input.MetadataRequestHandler_impl.handle(MetadataRequestHandler_impl.java:78)
at
org.apache.uima.adapter.jms.activemq.JmsInputChannel.onMessage(JmsInputChannel.java:728)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:535)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:952)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.uima.aae.UimaAsThreadFactory$1.run(UimaAsThreadFactory.java:129)
... 1 more
Caused by: org.apache.uima.analysis_engine.AnalysisEngineProcessException
at org.apache.uima.ducc.sampleapps.DuccCasCC.process(DuccCasCC.java:117)
at
org.apache.uima.analysis_component.JCasAnnotator_ImplBase.process(JCasAnnotator_ImplBase.java:48)
at
org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.callAnalysisComponentProcess(PrimitiveAnalysisEngine_impl.java:385)
at
org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.processAndOutputNewCASes(PrimitiveAnalysisEngine_impl.java:309)
at
org.apache.uima.analysis_engine.asb.impl.ASB_impl$AggregateCasIterator.processUntilNextOutputCas(ASB_impl.java:569)
at
org.apache.uima.analysis_engine.asb.impl.ASB_impl$AggregateCasIterator.init(ASB_impl.java:411)
at
org.apache.uima.analysis_engine.asb.impl.ASB_impl.process(ASB_impl.java:344)
at
org.apache.uima.analysis_engine.impl.AggregateAnalysisEngine_impl.processAndOutputNewCASes(AggregateAnalysisEngine_impl.java:266)
at
[ANNOUNCE] DKPro Core 1.7.0 released
We are pleased to announce the release of DKPro Core, version 1.7.0 (ASL GPL) a collection of interoperable software components for natural language processing (NLP) based on the Apache UIMA framework. . http://code.google.com/p/dkpro-core-asl . http://code.google.com/p/dkpro-core-gpl Analysis components . hunpos - wrapper for hunpos, a HMM pos tagger including models for many languages; . langdetect - wrapper for language-detection, a language detection tool for java; . mallet - wrapper for topic modelling using MALLET; . textnormalizer - original components for text normalization, e.g. spelling correction, umlaut normalization, expressive lengthening normalization. Data formats . io.conll - support for CoNLL 2000, 2002, 2009 and 2012 formats; . io.ditop - support for DiTop topic model visualization format; . io.penntree - support for combined and chunked formats; . io.tueppdz - support for TüPP-D/Z format. Further highlights in this release include: . Upgrade to Apache UIMA 2.6.0; . Upgrade LanguageTools to version 2.7; . Upgrade MaltParser to version 1.8; . Upgrade Stanford CoreNLP to version 3.4.1; . Support additional MaltParser models: Bengali, Farsi, Polish; . Support additional MSTParser models: Croatian; . Support additional OpenNLP models: Spanish; . Support additional Stanford CoreNLP models: Spanish, English caseless, shift-reduce parser models. A more detailed overview of the changes in this release can be found here: https://code.google.com/p/dkpro-core-asl/issues/list?can=1q=milestone%3D1.7.0colspec=ID+Type+Status+Priority+DKPro+Module+Milestone+Owner+Summarycells=tiles When upgrading, please mind that you should not mix different versions of DKPro Core components in your projects - they may not be compatible with each other. -- Pedro Santos, for the DKPro Core development team
Re: Ducc: Rename failed
To debug, please add the following option to the job submission: --all_in_one local This will run all the code in a single process on the machine doing the submit. Hopefully the log file and/or console will be more informative. On Fri, Nov 28, 2014 at 1:41 PM, Simon Hafner reactorm...@gmail.com wrote: 2014-11-28 10:45 GMT-06:00 Eddie Epstein eaepst...@gmail.com: DuccCasCC component has presumably created /home/ducc/analysis/txt.processed/5911.txt_0_processed.zip_temp and written to it? I don't know, the _temp file doesn't exist anymore. Did you run this sample job in something other than cluster mode? I get the same error running on a single machine.
Re: Ducc: Rename failed
2014-11-28 14:18 GMT-06:00 Eddie Epstein eaepst...@gmail.com: To debug, please add the following option to the job submission: --all_in_one local This will run all the code in a single process on the machine doing the submit. Hopefully the log file and/or console will be more informative. Yes, that helped. It was a missing classpath.
Re: DUCC doesn't use all available machines
Now you are hitting a limit configured in ducc.properties: # Max number of work-item CASes for each job ducc.threads.limit = 500 62 job process * 8 threads per process = 496 max concurrent work items. This was put in to limit the memory required by the job driver. This value can probably be pushed up in the range of 700-800 before the job driver will go OOM. There are configuration parameters to increase JD memory: # Memory size in MB allocated for each JD ducc.jd.share.quantum = 450 # JD max heap size. Should be smaller than the JD share quantum ducc.driver.jvm.args = -Xmx400M -DUimaAsCasTracking DUCC would have to be restarted for the JD size parameters to take effect. One of the current DUCC development items is to significantly reduce the memory needed per work item, and raise the default limit for concurrent work items by two or three orders of magnitude. On Fri, Nov 28, 2014 at 6:40 PM, Simon Hafner reactorm...@gmail.com wrote: I've put the fudge to 12000, and it jumped immediately to 62 procs. However, it doesn't spawn new ones even though it has about 6k items left and it doesn't spawn more procs. 2014-11-17 15:30 GMT-06:00 Jim Challenger chall...@gmail.com: It is also possible that RM prediction has decided that additional processes are not needed. It appears that there were likely 64 work items dispatched, plus the 6 completed, leaving only 30 that were idle. If these work items appeared to be completing quickly, the RM would decide that scale-up would be wasteful and not do it. Very gory details if you're interested: The time to start a new processes is measured by the RM based on the observed initialization time of the processes plus an estimate of how long it would take to get a new process actually running. A fudge-factor is added on top of this because in a large operation it is wasteful to start processes (with associated preemptions) that only end up doing a few work tems. All is subjective and configurable. The average time-per-work item is also reported to the RM. The RM then looks at the number of work items remaining, and the estimated time needed to processes this work based on the above, and if it determines that the job will be completed before new processes can be scaled up and initialized, it does not scale up. For short jobs, this can be a bit inaccurate, but those jobs are short :) For longer jobs, the time-per-work-item becomes increasingly accurate so the RM prediction tends to improve and ramp-up WILL occur if the work-item time turns out to be larger than originally thought. (Our experience is that work-item times are mostly uniform with occasional outliers, but the prediction seems to work well). Relevant configuration parameters in ducc.properties: # Predict when a job will end and avoid expanding if not needed. Set to false to disable prediction. ducc.rm.prediction = true # Add this fudge factor (milliseconds) to the expansion target when using prediction ducc.rm.prediction.fudge = 12 You can observe this in the rm log, see the example below. I'm preparing a guide to this log; for now, the net of these two log lines is: the projection for the job in question (job 208927) is that 16 processes are needed to complete this job, even though the job could use 20 processes at full expanseion - the BaseCap - so a max of 16 will be scheduled for it, subject to fair-share constraint. 17 Nov 2014 15:07:38,880 INFO RM.RmJob - */getPrjCap/* 208927 bobuser O 2 T 343171 NTh 128 TI 143171 TR 6748.601431980907 R 1.8967e-02 QR 5043 P 6509 F 0 ST 1416254363603*/return 16/* 17 Nov 2014 15:07:38,880 INFO RM.RmJob - */initJobCap/* 208927 bobuser O 2 */Base cap:/* 20 Expected future cap: 16 potential cap 16 actual cap 16 Jim On 11/17/14, 3:44 PM, Eddie Epstein wrote: DuccRawTextSpec.job specifies that each job process (JP) run 8 analytic pipeline threads. So for this job with 100 work items, no more than 13 JPs would ever be started. After successful initialization of the first JP, DUCC begins scaling up the number of JPs using doubling. During JP scale up the scheduler monitors the work item completion rate, compares that with the JP initialization time, and stops scaling up JPs when starting more JPs will not make the job run any faster. Of course JP scale up is also limited by the job's fair share of resources relative to total resources available for all preemptable jobs. To see more JPs, increase the number and/or size of the input text files, or decrease the number of pipeline threads per JP. Note that it can be counter productive to run too many pipeline threads per machine. Assuming analytic threads are 100% CPU bound, running more threads than real cores will often slow down the overall document processing rate. On Mon, Nov 17, 2014 at 6:48 AM, Simon Hafner
