If looks like user "mario" is issuing the ducc_cancel command: 19 May 2017 11:20:13,061 INFO OR.OrchestratorComponent - N/A stopJob id=104789 19 May 2017 11:20:13,062 INFO OR.OrchestratorComponent - 104789 isAuthorized mario is mario 19 May 2017 11:20:13,062 INFO OR.Reason - 104789 Reason user:mario role:role_user message:forced killed104789 killed104789 19 May 2017 11:20:13,066 INFO OR.StateJobAccounting - 104789 stateChange current[Completing] previous[Running] 19 May 2017 11:20:13,067 INFO OR.StateJobAccounting - 104789 complete CanceledByUser "forced killed104789 killed104789"
Is there a person or program that would have done so? Lou. On Fri, May 19, 2017 at 2:05 AM, priyank sharma <priyank.sha...@orkash.com> wrote: > Hey > > These are the orchestrator logs of the job when it was canceled by user. > > 19 May 2017 11:18:50,290 INFO OR.OrchestratorComponent - 104789 > reconcileJdState state: Active total: 4298 done: 3968 error: 0 killJob: > false > 19 May 2017 11:19:00,466 INFO OR.OrchestratorComponent - 104789 > reconcileJdState state: Active total: 4298 done: 3974 error: 0 killJob: > false > 19 May 2017 11:19:10,607 INFO OR.OrchestratorComponent - 104789 > reconcileJdState state: Active total: 4298 done: 3985 error: 0 killJob: > false > 19 May 2017 11:19:20,856 INFO OR.OrchestratorComponent - 104789 > reconcileJdState state: Active total: 4298 done: 3989 error: 0 killJob: > false > 19 May 2017 11:19:31,115 INFO OR.OrchestratorComponent - 104789 > reconcileJdState state: Active total: 4298 done: 3998 error: 0 killJob: > false > 19 May 2017 11:19:41,269 INFO OR.OrchestratorComponent - 104789 > reconcileJdState state: Active total: 4298 done: 4003 error: 0 killJob: > false > 19 May 2017 11:19:51,819 INFO OR.OrchestratorComponent - 104789 > reconcileJdState state: Active total: 4298 done: 4020 error: 0 killJob: > false > 19 May 2017 11:20:02,013 INFO OR.OrchestratorComponent - 104789 > reconcileJdState state: Active total: 4298 done: 4035 error: 0 killJob: > false > 19 May 2017 11:20:12,219 INFO OR.OrchestratorComponent - 104789 > reconcileJdState state: Active total: 4298 done: 4062 error: 0 killJob: > false > 19 May 2017 11:20:13,061 INFO OR.OrchestratorComponent - N/A stopJob > id=104789 > 19 May 2017 11:20:13,062 INFO OR.OrchestratorComponent - 104789 > isAuthorized mario is mario > 19 May 2017 11:20:13,062 INFO OR.Reason - 104789 Reason user:mario > role:role_user message:forced killed104789 killed104789 > 19 May 2017 11:20:13,066 INFO OR.StateJobAccounting - 104789 stateChange > current[Completing] previous[Running] > 19 May 2017 11:20:13,067 INFO OR.StateJobAccounting - 104789 complete > CanceledByUser "forced killed104789 killed104789" > 19 May 2017 11:20:13,067 INFO OR.ProcessAccounting - 104789 deallocate > 265 worker > 19 May 2017 11:20:13,067 INFO OR.ProcessAccounting - 104789 deallocate > 268 worker > 19 May 2017 11:20:13,068 INFO OR.ProcessAccounting - 104789 deallocate > 266 worker > 19 May 2017 11:20:13,068 INFO OR.ProcessAccounting - 104789 deallocate > 267 worker > 19 May 2017 11:20:13,068 INFO OR.ProcessAccounting - 104789 deallocate 0 > driver > 19 May 2017 11:20:13,068 INFO OR.OrchestratorCheckpoint - N/A saveState > saving to:/mario/apache-uima-ducc-2.0.1/state//orchestrator.ckpt > 19 May 2017 11:20:15,300 INFO OR.OrchestratorCheckpoint - N/A saveState > saved:/mario/apache-uima-ducc-2.0.1/state//orchestrator.ckpt > 19 May 2017 11:20:15,300 INFO OR.OrchestratorComponent - 104789 stopJob > job state:Completing > 19 May 2017 11:20:29,164 INFO OR.ProcessAccounting - 104789 > copyReasonForStoppingProcess 268 process reason code:Deallocated > 19 May 2017 11:20:29,164 INFO OR.ProcessAccounting - 104789 > copyProcessExitCode 268 process exit code:255 > 19 May 2017 11:20:29,167 INFO OR.ProcessAccounting - 104789 > copyReasonForStoppingProcess 266 process reason code:Deallocated > 19 May 2017 11:20:29,167 INFO OR.ProcessAccounting - 104789 > copyProcessExitCode 266 process exit code:255 > 19 May 2017 11:20:29,169 INFO OR.ProcessAccounting - 104789 > copyReasonForStoppingProcess 265 process reason code:Deallocated > 19 May 2017 11:20:29,170 INFO OR.ProcessAccounting - 104789 > copyProcessExitCode 265 process exit code:255 > 19 May 2017 11:20:29,171 INFO OR.ProcessAccounting - 104789 > copyReasonForStoppingProcess 267 process reason code:Deallocated > 19 May 2017 11:20:29,171 INFO OR.ProcessAccounting - 104789 > copyProcessExitCode 267 process exit code:255 > 19 May 2017 11:20:29,183 INFO OR.StateJobAccounting - 104789 stateChange > current[Completed] previous[Completing] > 19 May 2017 11:20:30,161 INFO OR.ProcessAccounting - 104789 > copyReasonForStoppingProcess 0 process reason code:KilledByDucc > 19 May 2017 11:20:30,162 INFO OR.ProcessAccounting - 104789 > copyProcessExitCode 0 process exit code:143 > 19 May 2017 11:20:34,081 INFO OR.OrchestratorComponent - N/A > assignDefaultFairShareClass scheduling_class=normal > 19 May 2017 11:20:34,091 WARN OR.JobFactory - N/A checkSpec unrecognized: > classpath > 19 May 2017 11:20:34,092 WARN OR.JobFactory - N/A checkSpec unrecognized: > environment > 19 May 2017 11:20:34,097 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.WorkItemTimeout=10 > 19 May 2017 11:20:34,097 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JobDirectory=/mario/ducc/logs/ > 19 May 2017 11:20:34,099 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpFlowController=org.apache.uima.ducc.FlowController > 19 May 2017 11:20:34,099 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpAeDescriptor=desc/orkash/ae/aggregate/Corefe > rnceAggDescriptor_SVO > 19 May 2017 11:20:34,099 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpAeOverrides=null > 19 May 2017 11:20:34,099 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpCcDescriptor=desc/orkash/cas_consumer/Elasti > cSearchCasConsumerDescriptor > 19 May 2017 11:20:34,100 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpCcOverrides=null > 19 May 2017 11:20:34,100 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpCmDescriptor=null > 19 May 2017 11:20:34,100 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpCmOverrides=null > 19 May 2017 11:20:34,100 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpDd=null > 19 May 2017 11:20:34,101 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpDdName=DUCC.Job > 19 May 2017 11:20:34,101 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpDdDescription=DUCC.Generated > 19 May 2017 11:20:34,101 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpThreadCount=5 > 19 May 2017 11:20:34,102 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpDdBrokerURL=${broker.name} > 19 May 2017 11:20:34,102 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.JpDdBrokerEndpoint=${queue.name} > 19 May 2017 11:20:34,102 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.UserErrorHandlerClassname=null > 19 May 2017 11:20:34,102 INFO OR.JobFactory - N/A addDashD > -Dducc.deploy.UserErrorHandlerCfg=null > 19 May 2017 11:20:34,103 INFO OR.JobFactory - 104791 createDriver driver > env vars: 3 > 19 May 2017 11:20:34,104 INFO OR.ProcessAccounting - 104791 addProcess 0 > added > 19 May 2017 11:20:34,104 INFO OR.JobFactory - N/A specification user:mario > 19 May 2017 11:20:34,104 INFO OR.JobFactory - N/A specification > signature:null > 19 May 2017 11:20:34,104 INFO OR.JobFactory - N/A specification > driver_descriptor_CR:desc/orkash/collection_reader/DBCollect > ionReaderMongoDBIdOnly > 19 May 2017 11:20:34,104 INFO OR.JobFactory - N/A specification > log_directory:/mario/ducc/logs/ > 19 May 2017 11:20:34,104 INFO OR.JobFactory - N/A specification > scheduling_class:normal > > Thanks and Regards > Priyank Sharma > > On Thursday 18 May 2017 07:29 PM, Lou DeGenaro wrote: > >> I'm still trying to image how your job got canceled "by user". There are >> just two ways, I'm pretty sure: you issued the ducc_cancel command or the >> cancel-on-interrupt flag was set and the ducc_submit command stopped heart >> beating. Do you have the orchestrator log file still? It should record >> the cancel request. >> >> With respect to standalone, can you visit http://<standalone-hostname>:4 >> 2133 >> and navigate to the System-->Daemons page? >> >> Lou. >> >> On Thu, May 18, 2017 at 9:38 AM, priyank sharma < >> priyank.sha...@orkash.com> >> wrote: >> >> Hey >>> >>> We have not specified the property "-cancel-on-interrupt" in the >>> ducc_submit script. >>> >>> Also, I tried to install a fresh copy of DUCC as a standalone server on >>> my >>> system and when I executed the command "start_ducc" it shows the >>> following >>> error: >>> >>> ActiveMQ broker is not running on tcp://user:61617 even though activemq >>> is >>> installed on the system. >>> >>> Thanks and Regards >>> Priyank Sharma >>> >>> On Thursday 18 May 2017 01:03 PM, Lou DeGenaro wrote: >>> >>> Priyank, >>>> >>>> You must have specified --cancel_on_interrupt when you submitted you >>>> job. >>>> This requires that the ducc_submit continue uninterrupted or else your >>>> job >>>> will be automatically canceled. >>>> >>>> The way this works is as follows: >>>> 1. you issue ducc_submit with the --cancel_on_interrupt flag >>>> 2. the ducc_submit CLI submits the job and continues to run sending >>>> heartbeats to ducc-mon to indicate that it is still alive >>>> 3. if the ducc_submit CLI is ctl-C'd or cannot contact the ducc-mon for >>>> 5 >>>> minutes your job is automatically canceled >>>> >>>> Be sure ducc_submit is still running. Be sure the machine on which >>>> ducc_submit is running can reach the machine where ducc-mon is running. >>>> As >>>> a stop-gap measure, you can submit the work without the >>>> --cancel_on_interrupt flag. >>>> >>>> Lou. >>>> >>>> On Thu, May 18, 2017 at 1:18 AM, priyank sharma < >>>> priyank.sha...@orkash.com> >>>> wrote: >>>> >>>> Hey Eddie >>>> >>>>> The job usually runs for over an hour before it is interrupted and >>>>> ultimately stopped due to cancelled by user. As seen in the logs, the >>>>> following message is displayed: >>>>> >>>>> completion type: CanceledByUser >>>>> rationale: "Terminate button pressed" >>>>> >>>>> There is no user interference in this, and the system is canceling the >>>>> job >>>>> itself. >>>>> >>>>> Thanks and Regards >>>>> Priyank Sharma >>>>> >>>>> On Wednesday 17 May 2017 06:57 PM, Eddie Epstein wrote: >>>>> >>>>> How long does the job run before stopping? Cancelled by user could come >>>>> >>>>>> if >>>>>> the job is submitted with cancel_on_interrupt and the client >>>>>> submitting >>>>>> the >>>>>> job were stopped. >>>>>> >>>>>> Eddie >>>>>> >>>>>> On Tue, May 16, 2017 at 8:31 AM, Lou DeGenaro <lou.degen...@gmail.com >>>>>> > >>>>>> wrote: >>>>>> >>>>>> Dunno why the connection would be refused. Are the JD and JP on the >>>>>> same >>>>>> >>>>>> or different machines? Is the network viable between the machines on >>>>>>> which >>>>>>> each is located? >>>>>>> >>>>>>> Lou. >>>>>>> >>>>>>> On Tue, May 16, 2017 at 8:18 AM, priyank sharma < >>>>>>> priyank.sha...@orkash.com >>>>>>> wrote: >>>>>>> >>>>>>> Hey! >>>>>>> >>>>>>> There were no error found in JD log.Following is a snippet of the jD >>>>>>>> log >>>>>>>> >>>>>>>> 14 May 2017 18:47:39,593 INFO ActionGet - T[482] engage seqNo=3484 >>>>>>>> remote=S144.3170.35 >>>>>>>> 14 May 2017 18:47:39,641 INFO ActionGet - T[283] engage seqNo=3485 >>>>>>>> remote=S144.2443.34 >>>>>>>> 14 May 2017 18:47:40,688 INFO ActionEnd - T[284] engage seqNo=3470 >>>>>>>> remote=S144.2443.36 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:47:40,736 INFO ActionGet - T[483] engage seqNo=3486 >>>>>>>> remote=S144.2443.36 >>>>>>>> 14 May 2017 18:47:43,207 INFO ActionEnd - T[482] engage seqNo=3477 >>>>>>>> remote=S144.3346.32 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:47:43,254 INFO ActionGet - T[284] engage seqNo=3487 >>>>>>>> remote=S144.3346.32 >>>>>>>> 14 May 2017 18:47:43,258 INFO ActionEnd - T[283] engage seqNo=3467 >>>>>>>> remote=S144.2443.35 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:47:43,296 INFO ActionGet - T[483] engage seqNo=3488 >>>>>>>> remote=S144.2443.35 >>>>>>>> 14 May 2017 18:47:44,425 INFO ActionEnd - T[283] engage seqNo=3468 >>>>>>>> remote=S144.3346.34 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:47:44,605 INFO ActionGet - T[483] engage seqNo=3489 >>>>>>>> remote=S144.3346.34 >>>>>>>> 14 May 2017 18:47:46,105 INFO ActionEnd - T[283] engage seqNo=3480 >>>>>>>> remote=S144.3346.33 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:47:46,166 INFO ActionGet - T[482] engage seqNo=3490 >>>>>>>> remote=S144.3346.33 >>>>>>>> 14 May 2017 18:47:46,233 INFO ActionEnd - T[284] engage seqNo=3478 >>>>>>>> remote=S144.3346.36 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:47:46,415 INFO ActionGet - T[482] engage seqNo=3491 >>>>>>>> remote=S144.3346.36 >>>>>>>> 14 May 2017 18:47:49,924 INFO ActionEnd - T[284] engage seqNo=3475 >>>>>>>> remote=S144.3348.35 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:47:49,968 INFO ActionGet - T[482] engage seqNo=3492 >>>>>>>> remote=S144.3348.35 >>>>>>>> 14 May 2017 18:47:50,856 INFO ActionEnd - T[283] engage seqNo=3469 >>>>>>>> remote=S144.3348.32 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:47:50,918 INFO ActionGet - T[284] engage seqNo=3493 >>>>>>>> remote=S144.3348.32 >>>>>>>> 14 May 2017 18:47:53,566 INFO ActionEnd - T[284] engage seqNo=3459 >>>>>>>> remote=S144.2443.33 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:47:53,599 INFO ActionGet - T[483] engage seqNo=3494 >>>>>>>> remote=S144.2443.33 >>>>>>>> 14 May 2017 18:47:58,507 INFO ActionEnd - T[283] engage seqNo=3473 >>>>>>>> remote=S144.3348.36 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:47:58,565 INFO ActionGet - T[284] engage seqNo=3495 >>>>>>>> remote=S144.3348.36 >>>>>>>> 14 May 2017 18:48:06,218 INFO ActionEnd - T[283] engage seqNo=3460 >>>>>>>> remote=S144.3348.34 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:06,360 INFO ActionGet - T[483] engage seqNo=3496 >>>>>>>> remote=S144.3348.34 >>>>>>>> 14 May 2017 18:48:09,619 INFO ActionEnd - T[283] engage seqNo=3481 >>>>>>>> remote=S144.2443.32 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:09,674 INFO ActionEnd - T[483] engage seqNo=3479 >>>>>>>> remote=S144.3170.36 ended >>>>>>>> 14 May 2017 18:48:09,681 INFO ActionGet - T[284] engage seqNo=3497 >>>>>>>> remote=S144.2443.32 >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:09,814 INFO ActionGet - T[482] engage seqNo=3498 >>>>>>>> remote=S144.3170.36 >>>>>>>> 14 May 2017 18:48:13,464 INFO ActionEnd - T[283] engage seqNo=3476 >>>>>>>> remote=S144.3346.35 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:13,498 INFO ActionGet - T[483] engage seqNo=3499 >>>>>>>> remote=S144.3346.35 >>>>>>>> 14 May 2017 18:48:15,116 INFO ActionEnd - T[284] engage seqNo=3482 >>>>>>>> remote=S144.3170.32 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:15,163 INFO ActionGet - T[283] engage seqNo=3500 >>>>>>>> remote=S144.3170.32 >>>>>>>> 14 May 2017 18:48:17,050 INFO ActionEnd - T[284] engage seqNo=3465 >>>>>>>> remote=S144.3170.33 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:17,141 INFO ActionGet - T[482] engage seqNo=3501 >>>>>>>> remote=S144.3170.33 >>>>>>>> 14 May 2017 18:48:19,138 INFO ActionEnd - T[284] engage seqNo=3471 >>>>>>>> remote=S144.3170.34 ended >>>>>>>> 14 May 2017 18:48:19,148 INFO ActionEnd - T[283] engage seqNo=3487 >>>>>>>> remote=S144.3346.32 ended >>>>>>>> in getNext >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:19,180 INFO ActionGet - T[483] engage seqNo=3502 >>>>>>>> remote=S144.3170.34 >>>>>>>> 14 May 2017 18:48:19,262 INFO ActionGet - T[284] engage seqNo=3503 >>>>>>>> remote=S144.3346.32 >>>>>>>> 14 May 2017 18:48:22,923 INFO ActionEnd - T[482] engage seqNo=3486 >>>>>>>> remote=S144.2443.36 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:22,977 INFO ActionGet - T[284] engage seqNo=3504 >>>>>>>> remote=S144.2443.36 >>>>>>>> 14 May 2017 18:48:32,013 INFO ActionEnd - T[284] engage seqNo=3492 >>>>>>>> remote=S144.3348.35 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:32,055 INFO ActionGet - T[483] engage seqNo=3505 >>>>>>>> remote=S144.3348.35 >>>>>>>> 14 May 2017 18:48:34,053 INFO ActionEnd - T[284] engage seqNo=3501 >>>>>>>> remote=S144.3170.33 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:34,145 INFO ActionGet - T[483] engage seqNo=3506 >>>>>>>> remote=S144.3170.33 >>>>>>>> 14 May 2017 18:48:36,116 INFO ActionEnd - T[483] engage seqNo=3485 >>>>>>>> remote=S144.2443.34 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:36,156 INFO ActionGet - T[482] engage seqNo=3507 >>>>>>>> remote=S144.2443.34 >>>>>>>> 14 May 2017 18:48:37,736 INFO ActionEnd - T[284] engage seqNo=3488 >>>>>>>> remote=S144.2443.35 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:37,770 INFO ActionEnd - T[483] engage seqNo=3484 >>>>>>>> remote=S144.3170.35 ended >>>>>>>> 14 May 2017 18:48:37,776 INFO ActionGet - T[283] engage seqNo=3508 >>>>>>>> remote=S144.2443.35 >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:37,834 INFO ActionGet - T[482] engage seqNo=3509 >>>>>>>> remote=S144.3170.35 >>>>>>>> 14 May 2017 18:48:40,161 INFO ActionEnd - T[483] engage seqNo=3490 >>>>>>>> remote=S144.3346.33 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:40,256 INFO ActionGet - T[482] engage seqNo=3510 >>>>>>>> remote=S144.3346.33 >>>>>>>> 14 May 2017 18:48:44,891 INFO ActionEnd - T[284] engage seqNo=3493 >>>>>>>> remote=S144.3348.32 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:48:44,929 INFO ActionGet - T[483] engage seqNo=3511 >>>>>>>> remote=S144.3348.32 >>>>>>>> 14 May 2017 18:49:02,007 INFO ActionEnd - T[483] engage seqNo=3489 >>>>>>>> remote=S144.3346.34 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:49:02,086 INFO ActionGet - T[283] engage seqNo=3512 >>>>>>>> remote=S144.3346.34 >>>>>>>> 14 May 2017 18:49:03,407 INFO ActionEnd - T[283] engage seqNo=3502 >>>>>>>> remote=S144.3170.34 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:49:03,439 INFO ActionGet - T[482] engage seqNo=3513 >>>>>>>> remote=S144.3170.34 >>>>>>>> 14 May 2017 18:49:04,963 INFO ActionEnd - T[482] engage seqNo=3498 >>>>>>>> remote=S144.3170.36 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:49:05,010 INFO ActionGet - T[284] engage seqNo=3514 >>>>>>>> remote=S144.3170.36 >>>>>>>> 14 May 2017 18:49:06,442 INFO ActionEnd - T[284] engage seqNo=3495 >>>>>>>> remote=S144.3348.36 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:49:06,501 INFO ActionGet - T[483] engage seqNo=3515 >>>>>>>> remote=S144.3348.36 >>>>>>>> 14 May 2017 18:49:07,690 INFO ActionEnd - T[284] engage seqNo=3500 >>>>>>>> remote=S144.3170.32 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:49:07,730 INFO ActionGet - T[483] engage seqNo=3516 >>>>>>>> remote=S144.3170.32 >>>>>>>> 14 May 2017 18:49:08,734 INFO ActionEnd - T[284] engage seqNo=3497 >>>>>>>> remote=S144.2443.32 ended >>>>>>>> 14 May 2017 18:49:08,757 INFO ActionEnd - T[283] engage seqNo=3496 >>>>>>>> remote=S144.3348.34 ended >>>>>>>> in getNext >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:49:08,792 INFO ActionGet - T[483] engage seqNo=3517 >>>>>>>> remote=S144.2443.32 >>>>>>>> 14 May 2017 18:49:08,874 INFO ActionGet - T[482] engage seqNo=3518 >>>>>>>> remote=S144.3348.34 >>>>>>>> 14 May 2017 18:49:10,904 INFO ActionEnd - T[284] engage seqNo=3510 >>>>>>>> remote=S144.3346.33 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:49:10,952 INFO ActionGet - T[283] engage seqNo=3519 >>>>>>>> remote=S144.3346.33 >>>>>>>> 14 May 2017 18:49:12,970 INFO ActionEnd - T[482] engage seqNo=3504 >>>>>>>> remote=S144.2443.36 ended >>>>>>>> in getNext >>>>>>>> 14 May 2017 18:49:13,022 INFO ActionGet - T[284] engage seqNo=3520 >>>>>>>> remote=S144.2443.36 >>>>>>>> >>>>>>>> >>>>>>>> Thanks and Regards >>>>>>>> Priyank Sharma >>>>>>>> >>>>>>>> >>>>>>>> On Tuesday 16 May 2017 04:41 PM, Lou DeGenaro wrote: >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> There are two parts: JP (one or more) and JD (one). You have shown >>>>>>>>> the >>>>>>>>> log >>>>>>>>> from a JP, which is trying to contact the JD for more work. Can >>>>>>>>> you >>>>>>>>> >>>>>>>>> share >>>>>>>>> >>>>>>>> the JD log? >>>>>>>> >>>>>>>> Also, you can find me on HipChat https://apache.hipchat.com/cha >>>>>>>>> t/room/3665278 >>>>>>>>> in about an hour from now. >>>>>>>>> >>>>>>>>> Lou. >>>>>>>>> >>>>>>>>> On Tue, May 16, 2017 at 2:04 AM, priyank sharma < >>>>>>>>> priyank.sha...@orkash.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hey >>>>>>>>> >>>>>>>>> I was running the Ducc job with the batch of around 4000 document. >>>>>>>>> It >>>>>>>>> >>>>>>>>>> was >>>>>>>>>> >>>>>>>>> able to ingest around 3000 document but after that it automatically >>>>>>>> >>>>>>>> stopped >>>>>>>>> >>>>>>>>>> and gave the Reason or extraordinary status as canceled by user. >>>>>>>>>> Then >>>>>>>>>> >>>>>>>>>> it >>>>>>>>>> >>>>>>>>> started the new job with the same batch, and it has been going on >>>>>>>> in >>>>>>>> >>>>>>>> the >>>>>>>>> >>>>>>>>> same manner. >>>>>>>> >>>>>>>> As checked in the logs the following error was found:- >>>>>>>>> >>>>>>>>>> java.net.ConnectException: Connection refused >>>>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native >>>>>>>>>> Method) >>>>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>>> >>>>>>>>>> doConnect(AbstractPlainSock >>>>>>>>>> >>>>>>>>> etImpl.java:339) >>>>>>>> >>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>> >>>>>>>>>> connectToAddress(AbstractPl >>>>>>>>>> >>>>>>>>> ainSocketImpl.java:200) >>>>>>>> >>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>> >>>>>>>>>> connect(AbstractPlainSocket >>>>>>>>>> >>>>>>>>> Impl.java:182) >>>>>>>> >>>>>>>> at java.net.SocksSocketImpl.conne >>>>>>>>> >>>>>>>>>> ct(SocksSocketImpl.java:392) >>>>>>>>>> at java.net.Socket.connect(Socket.java:579) >>>>>>>>>> at java.net.Socket.connect(Socket.java:528) >>>>>>>>>> at java.net.Socket.<init>(Socket.java:425) >>>>>>>>>> at java.net.Socket.<init>(Socket.java:280) >>>>>>>>>> at org.apache.commons.httpclient. >>>>>>>>>> >>>>>>>>>> protocol.DefaultProtocolSocket >>>>>>>>>> >>>>>>>>> Factory.createSocket(DefaultProtocolSocketFactory.java:80) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> protocol.DefaultProtocolSocket >>>>>>>>>> >>>>>>>>> Factory.createSocket(DefaultProtocolSocketFactory.java:122) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpConnection.open(HttpConnec >>>>>>>>>> >>>>>>>>> tion.java:707) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> MultiThreadedHttpConnectionMan >>>>>>>>>> >>>>>>>>> ager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionM >>>>>>>> >>>>>>>> anager.java:1361) >>>>>>>>> >>>>>>>>>> at org.apache.commons.httpclient. >>>>>>>>>> >>>>>>>>>> HttpMethodDirector.executeWith >>>>>>>>>> >>>>>>>>> Retry(HttpMethodDirector.java:387) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpMethodDirector.executeMeth >>>>>>>>>> >>>>>>>>> od(HttpMethodDirector.java:171) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpClient.executeMethod(HttpC >>>>>>>>>> >>>>>>>>> lient.java:397) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpClient.executeMethod(HttpC >>>>>>>>>> >>>>>>>>> lient.java:323) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> DuccHttpClie >>>>>>>>>> >>>>>>>>> nt.execute(DuccHttpClient.java:217) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> HttpWorkerTh >>>>>>>>>> >>>>>>>>> read.run(HttpWorkerThread.java:287) >>>>>>>> >>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>>>>>>> >>>>>>>>>> Executors.java:471) >>>>>>>>>> at java.util.concurrent.FutureTas >>>>>>>>>> k.run(FutureTask.java:262) >>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>>>>>>> >>>>>>>>>> ThreadPool >>>>>>>>>> >>>>>>>>> Executor.java:1145) >>>>>>>> >>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>>>>>> >>>>>>>>>> ThreadPoo >>>>>>>>>> >>>>>>>>> lExecutor.java:615) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> UimaServiceT >>>>>>>>>> >>>>>>>>> hreadFactory$1.run(UimaServiceThreadFactory.java:85) >>>>>>>> >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> >>>>>>>>>> 15 May 2017 16:18:23,760 ERROR DuccHttpClient - T[36] run >>>>>>>>>> java.net.ConnectException: Connection refused >>>>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native >>>>>>>>>> Method) >>>>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>>> >>>>>>>>>> doConnect(AbstractPlainSock >>>>>>>>>> >>>>>>>>> etImpl.java:339) >>>>>>>> >>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>> >>>>>>>>>> connectToAddress(AbstractPl >>>>>>>>>> >>>>>>>>> ainSocketImpl.java:200) >>>>>>>> >>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>> >>>>>>>>>> connect(AbstractPlainSocket >>>>>>>>>> >>>>>>>>> Impl.java:182) >>>>>>>> >>>>>>>> at java.net.SocksSocketImpl.conne >>>>>>>>> >>>>>>>>>> ct(SocksSocketImpl.java:392) >>>>>>>>>> at java.net.Socket.connect(Socket.java:579) >>>>>>>>>> at java.net.Socket.connect(Socket.java:528) >>>>>>>>>> at java.net.Socket.<init>(Socket.java:425) >>>>>>>>>> at java.net.Socket.<init>(Socket.java:280) >>>>>>>>>> at org.apache.commons.httpclient. >>>>>>>>>> >>>>>>>>>> protocol.DefaultProtocolSocket >>>>>>>>>> >>>>>>>>> Factory.createSocket(DefaultProtocolSocketFactory.java:80) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> protocol.DefaultProtocolSocket >>>>>>>>>> >>>>>>>>> Factory.createSocket(DefaultProtocolSocketFactory.java:122) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpConnection.open(HttpConnec >>>>>>>>>> >>>>>>>>> tion.java:707) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> MultiThreadedHttpConnectionMan >>>>>>>>>> >>>>>>>>> ager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionM >>>>>>>> >>>>>>>> anager.java:1361) >>>>>>>>> >>>>>>>>>> at org.apache.commons.httpclient. >>>>>>>>>> >>>>>>>>>> HttpMethodDirector.executeWith >>>>>>>>>> >>>>>>>>> Retry(HttpMethodDirector.java:387) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpMethodDirector.executeMeth >>>>>>>>>> >>>>>>>>> od(HttpMethodDirector.java:171) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpClient.executeMethod(HttpC >>>>>>>>>> >>>>>>>>> lient.java:397) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpClient.executeMethod(HttpC >>>>>>>>>> >>>>>>>>> lient.java:323) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> DuccHttpClie >>>>>>>>>> >>>>>>>>> nt.execute(DuccHttpClient.java:217) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> HttpWorkerTh >>>>>>>>>> >>>>>>>>> read.run(HttpWorkerThread.java:287) >>>>>>>> >>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>>>>>>> >>>>>>>>>> Executors.java:471) >>>>>>>>>> at java.util.concurrent.FutureTas >>>>>>>>>> k.run(FutureTask.java:262) >>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>>>>>>> >>>>>>>>>> ThreadPool >>>>>>>>>> >>>>>>>>> Executor.java:1145) >>>>>>>> >>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>>>>>> >>>>>>>>>> ThreadPoo >>>>>>>>>> >>>>>>>>> lExecutor.java:615) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> UimaServiceT >>>>>>>>>> >>>>>>>>> hreadFactory$1.run(UimaServiceThreadFactory.java:85) >>>>>>>> >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> >>>>>>>>>> 15 May 2017 16:18:23,760 ERROR HttpWorkerThread - T[36] run >>>>>>>>>> java.net.ConnectException: Connection refused >>>>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native >>>>>>>>>> Method) >>>>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>>> >>>>>>>>>> doConnect(AbstractPlainSock >>>>>>>>>> >>>>>>>>> etImpl.java:339) >>>>>>>> >>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>> >>>>>>>>>> connectToAddress(AbstractPl >>>>>>>>>> >>>>>>>>> ainSocketImpl.java:200) >>>>>>>> >>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>> >>>>>>>>>> connect(AbstractPlainSocket >>>>>>>>>> >>>>>>>>> Impl.java:182) >>>>>>>> >>>>>>>> at java.net.SocksSocketImpl.conne >>>>>>>>> >>>>>>>>>> ct(SocksSocketImpl.java:392) >>>>>>>>>> at java.net.Socket.connect(Socket.java:579) >>>>>>>>>> at java.net.Socket.connect(Socket.java:528) >>>>>>>>>> at java.net.Socket.<init>(Socket.java:425) >>>>>>>>>> at java.net.Socket.<init>(Socket.java:280) >>>>>>>>>> at org.apache.commons.httpclient. >>>>>>>>>> >>>>>>>>>> protocol.DefaultProtocolSocket >>>>>>>>>> >>>>>>>>> Factory.createSocket(DefaultProtocolSocketFactory.java:80) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> protocol.DefaultProtocolSocket >>>>>>>>>> >>>>>>>>> Factory.createSocket(DefaultProtocolSocketFactory.java:122) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpConnection.open(HttpConnec >>>>>>>>>> >>>>>>>>> tion.java:707) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> MultiThreadedHttpConnectionMan >>>>>>>>>> >>>>>>>>> ager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionM >>>>>>>> >>>>>>>> anager.java:1361) >>>>>>>>> >>>>>>>>>> at org.apache.commons.httpclient. >>>>>>>>>> >>>>>>>>>> HttpMethodDirector.executeWith >>>>>>>>>> >>>>>>>>> Retry(HttpMethodDirector.java:387) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpMethodDirector.executeMeth >>>>>>>>>> >>>>>>>>> od(HttpMethodDirector.java:171) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpClient.executeMethod(HttpC >>>>>>>>>> >>>>>>>>> lient.java:397) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpClient.executeMethod(HttpC >>>>>>>>>> >>>>>>>>> lient.java:323) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> DuccHttpClie >>>>>>>>>> >>>>>>>>> nt.execute(DuccHttpClient.java:217) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> HttpWorkerTh >>>>>>>>>> >>>>>>>>> read.run(HttpWorkerThread.java:287) >>>>>>>> >>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>>>>>>> >>>>>>>>>> Executors.java:471) >>>>>>>>>> at java.util.concurrent.FutureTas >>>>>>>>>> k.run(FutureTask.java:262) >>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>>>>>>> >>>>>>>>>> ThreadPool >>>>>>>>>> >>>>>>>>> Executor.java:1145) >>>>>>>> >>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>>>>>> >>>>>>>>>> ThreadPoo >>>>>>>>>> >>>>>>>>> lExecutor.java:615) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> UimaServiceT >>>>>>>>>> >>>>>>>>> hreadFactory$1.run(UimaServiceThreadFactory.java:85) >>>>>>>> >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> >>>>>>>>>> java.net.ConnectException: Connection refused >>>>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native >>>>>>>>>> Method) >>>>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>>> >>>>>>>>>> doConnect(AbstractPlainSock >>>>>>>>>> >>>>>>>>> etImpl.java:339) >>>>>>>> >>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>> >>>>>>>>>> connectToAddress(AbstractPl >>>>>>>>>> >>>>>>>>> ainSocketImpl.java:200) >>>>>>>> >>>>>>>> at java.net.AbstractPlainSocketImpl. >>>>>>>>> >>>>>>>>>> connect(AbstractPlainSocket >>>>>>>>>> >>>>>>>>> Impl.java:182) >>>>>>>> >>>>>>>> at java.net.SocksSocketImpl.conne >>>>>>>>> >>>>>>>>>> ct(SocksSocketImpl.java:392) >>>>>>>>>> at java.net.Socket.connect(Socket.java:579) >>>>>>>>>> at java.net.Socket.connect(Socket.java:528) >>>>>>>>>> at java.net.Socket.<init>(Socket.java:425) >>>>>>>>>> at java.net.Socket.<init>(Socket.java:280) >>>>>>>>>> at org.apache.commons.httpclient. >>>>>>>>>> >>>>>>>>>> protocol.DefaultProtocolSocket >>>>>>>>>> >>>>>>>>> Factory.createSocket(DefaultProtocolSocketFactory.java:80) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> protocol.DefaultProtocolSocket >>>>>>>>>> >>>>>>>>> Factory.createSocket(DefaultProtocolSocketFactory.java:122) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpConnection.open(HttpConnec >>>>>>>>>> >>>>>>>>> tion.java:707) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> MultiThreadedHttpConnectionMan >>>>>>>>>> >>>>>>>>> ager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionM >>>>>>>> >>>>>>>> anager.java:1361) >>>>>>>>> >>>>>>>>>> at org.apache.commons.httpclient. >>>>>>>>>> >>>>>>>>>> HttpMethodDirector.executeWith >>>>>>>>>> >>>>>>>>> Retry(HttpMethodDirector.java:387) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpMethodDirector.executeMeth >>>>>>>>>> >>>>>>>>> od(HttpMethodDirector.java:171) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpClient.executeMethod(HttpC >>>>>>>>>> >>>>>>>>> lient.java:397) >>>>>>>> >>>>>>>> at org.apache.commons.httpclient. >>>>>>>>> >>>>>>>>>> HttpClient.executeMethod(HttpC >>>>>>>>>> >>>>>>>>> lient.java:323) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> DuccHttpClie >>>>>>>>>> >>>>>>>>> nt.execute(DuccHttpClient.java:217) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> HttpWorkerTh >>>>>>>>>> >>>>>>>>> read.run(HttpWorkerThread.java:287) >>>>>>>> >>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call( >>>>>>>>> >>>>>>>>>> Executors.java:471) >>>>>>>>>> at java.util.concurrent.FutureTas >>>>>>>>>> k.run(FutureTask.java:262) >>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>>>>>>> >>>>>>>>>> ThreadPool >>>>>>>>>> >>>>>>>>> Executor.java:1145) >>>>>>>> >>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>>>>>> >>>>>>>>>> ThreadPoo >>>>>>>>>> >>>>>>>>> lExecutor.java:615) >>>>>>>> >>>>>>>> at org.apache.uima.ducc.transport.configuration.jp. >>>>>>>>> >>>>>>>>>> UimaServiceT >>>>>>>>>> >>>>>>>>> hreadFactory$1.run(UimaServiceThreadFactory.java:85) >>>>>>>> >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> >>>>>>>>>> Exiting Process Due to a Framework error >>>>>>>>>> 15 May 2017 16:18:23,761 ERROR HttpWorkerThread - T[36] run The >>>>>>>>>> Job >>>>>>>>>> Process Terminating Due To a Framework Error >>>>>>>>>> >>>>>>>>>> Please reply as soon as possible. >>>>>>>>>> Thanks in advance. >>>>>>>>>> >>>>>>>>>> Priyank Sharma >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >