Re: DUCC's job goes into infintie loop

2017-11-15 Thread Lou DeGenaro
I'm sorry, but it is still not clear to me. You need to give step by step instructions on precisely what you are doing and the events that occur so I can re-create the problem, and/or you need to supply DUCC daemons and user logs and say which Job or Service number is not behaving properly. The b

Re: DUCC's job goes into infintie loop

2017-11-15 Thread priyank sharma
My complaint is that the job doesn't able to process any data as we gave maximum 75 minutes for a job to process, after every 75 minutes a new job starts it is having the same batch of ids as the previous job and this behaviour of the job continues untill we restarts the DUCC. Last time when t

Re: DUCC's job goes into infintie loop

2017-11-15 Thread Lou DeGenaro
Please note that we make a clear distinction between "services" and "jobs". Earlier e-mail from you suggested that your trouble was related to jobs. Here is my understanding of you situation. You use ducc_submit to submit a job comprising several work items. DUCC starts three Job Processes all

Re: DUCC's job goes into infintie loop

2017-11-14 Thread priyank sharma
server down mean one out of three machine is disconnected from the cluster of three and all the services were deployed on the machine which was disconnect from the cluster. Thanks and Regards Priyank Sharma On Tuesday 14 November 2017 04:08 PM, Lou DeGenaro wrote: What do you mean by "server

Re: DUCC's job goes into infintie loop

2017-11-14 Thread Lou DeGenaro
What do you mean by "server down", precisely? Since we have no logs to look at we can only go by your descriptions. We're trying to help... Lou. On Mon, Nov 13, 2017 at 11:30 PM, priyank sharma wrote: > When our job goes into infinite-loop that time uima analysis engine did > not start and on

Re: DUCC's job goes into infintie loop

2017-11-13 Thread priyank sharma
When our job goes into infinite-loop that time uima analysis engine did not start and one of the server out of three were down that server has all the service which is being used by the uima analysis engine. Is the server down creates this issue? is memory the problem? Thanks and Regards Priy

Re: DUCC's job goes into infintie loop

2017-11-13 Thread Eddie Epstein
Several different issues here. There is no "job completion cap", rather there is a limit on how long an individual work item will be allowed to process before it is labeled a timeout. The default number of such errors + exceptions before a Job is stopped is 15. Please increase this cap if you expec

Re: DUCC's job goes into infintie loop

2017-11-12 Thread priyank sharma
Yes, i am using DUCC v2.0.1 i have a three node cluster with 32gb ram, 40gb ram and 28gb ram. Job runs fine for 15-20 days after that it goes into the infinite loop with the same batch of the id's. We have a 75 minutes cap for a job to complete if not then its start again so after every 75 minu

Re: DUCC's job goes into infintie loop

2017-11-10 Thread Eddie Epstein
Hi Priyank, Looks like you are running DUCC v2.0.x. There are so many bugs fixed in subsequent versions, the latest being v2.2.1. Newer versions have a ducc_update command that will upgrade an existing install, but given all the changes since v2.0.x I suggest a clean install. Eddie On Fri, Nov 1

Re: DUCC's job goes into infintie loop

2017-11-10 Thread Lou DeGenaro
Are you running with a shared file system on your cluster? Is your user log directory located there? Look at the DUCC daemon log files located in $DUCC_HOME/logs. They should provide some clues as to what is wrong. Feel free to post (non-confidential versions of) them here for a second opinion.

Re: DUCC's job goes into infintie loop

2017-11-09 Thread priyank sharma
There is nothing on the work item page and performance page on the web server. There is only one log file for the main node, no log files for other two nodes. Ducc job processes not able to pick the data from the data source and no UIMA aggregator is working for that batches. Are the issue bec

Re: DUCC's job goes into infintie loop

2017-11-09 Thread Lou DeGenaro
The first place to look is in your job's logs. Visit the ducc-mon jobs page ducchost:42133/jobs.jsp then click on the id of your job. Examine the logs by clicking on each log file name looking for any revealing information. Feel free to post non-confidential snippets here, or If you'd like to ch

DUCC's job goes into infintie loop

2017-11-09 Thread priyank sharma
All! I have a problem regarding DUCC cluster in which a job process gets stuck and keeps on processing the same batch again and again due to maximum duration the batch gets reason or extraordinary status *"**CanceledByUser" *and then gets restarted with the same ID's. This usually happens aft