I just found this page:
http://www.globus.org/api/c-globus-4.0/globus_scheduler_event_generator/html/group__seg__api.html
So it *is* a failure code... wow. Okay, now can someone point me to
a list of such failure codes so I can choose something more generic
or appropriate than what we're currently using? Thanks again,
Adam
On Fri, Apr 4, 2008 at 10:39 AM, Adam Bazinet
<[EMAIL PROTECTED]> wrote:
Thanks for the replies, guys. I think it is the case that there is
an actual underlying job failure (which is my fault for not finding
earlier) and that the error message is a complete red herring.
Basically, we have a SEG that we wrote from scratch for BOINC jobs,
and in the case that a job fails for some reason the following code
gets executed:
//job failed
time_t ts;
globus_scheduler_event_failed(ts, wu_name, 5);
fprintf(fp, "%s : Set %s to failed\n",asctime(t), wu_name);
break;
Now, it's my guess that the "Invalid executable path" error
generated before is just some kind of default error message, mainly
because I don't see any place in our code where we specify the error
message, unless that "5" as the last argument to
globus_scheduler_event_failed means something specific. Does anyone
know what that last argument means? I looked quickly here (http://www.globus.org/toolkit/docs/development/3.9.4/execution/wsgram/developer/scheduler-tutorial-seg.html
) but didn't see it.
Thanks,
Adam
On Wed, Apr 2, 2008 at 9:54 AM, Stuart Martin <[EMAIL PROTECTED]>
wrote:
Adam,
If we are dealing with a tmp dir, a cluster, and many jobs/lots of
activity on the cluster, this could be a problem with the shared
file system. Maybe occasionally, the compute host where the job is
run cannot see the tmp dir / executable?
-Stu
On Apr 1, 2008, at Apr 1, 11:41 PM, [EMAIL PROTECTED] wrote:
Hi Adam,
i can't say much about it right now, but at first glance it looks
to me that the application causes the problem. I have to add that
i don't know BOINC and probably i didn't understand all details.
Can you describe the role of hmmpfam a bit more:
* this is not the main executable, right?
* it is called by the main executable somehow under certain
conditions?
* if so: you said that hmmpfam actually should not be used at all.
in what situation could the executable call hmmpfam (i love
that word ... :-) ).
Martin
Hi,
We are experiencing a strange problem that is causing jobs to fail,
albeit
somewhat randomly and infrequently. At any given time, we may have
100
active GRAM jobs on a given resource. All of these jobs submit fine
and
there are usually no immediate failures. However, every so often, one
will
fail, and this can be days after it was submitted, with the following
error:
[EMAIL PROTECTED]:/export/grid_files/260600020.09477316738932795>
globusrun-ws -status -j jobEPR.txt
Current job state: Failed
globusrun-ws: Job failed: Invalid executable path
"/export/scratch/applications/a5671f0138bc65dc700001aa80a3f378/
hmmpfam".
ProcessDied
Now, the resource we are submitting to is unique in that we are not
actually
transferring in the hmmpfam executable; that is just a dummy path, and
our
custom BOINC job manager does not attempt to make use of it, as BOINC
executables live elsewhere. So until recently, the executable
specified
on
that path never existed, and so the error *kinda* made sense; what
didn't
make sense is why it happened randomly. In an attempt to make this
problem
go away, I now have the BOINC job manager create a dummy executable
on that
path when the job is submitted, but it doesn't look like that has
helped
because the error is still popping up. *Now* the error message
certainly
doesn't make sense if taken at face value, because that has been a
"valid
path", technically speaking, for the lifetime of the job in question
-- yet
the job still failed. If it helps, I'll attach debug output below,
though
I
wasn't able to glean any additional information from it. Does anyone
have
a
guess as to why this would happen so randomly and infrequently, or
happen
in
the first place?
This one is costing us big time because when a job fails, Globus
deletes
all
the output collected thus far, and these are large batches of work.
Thanks!
Adam
[EMAIL PROTECTED]:/export/grid_files/260600020.09477316738932795>
globusrun-ws -debug -status -j jobEPR.txt
=== REQUEST MESSAGE (length 816) (time 1206627634.506184000) ===
<ns00:Envelope
xmlns:ns00="http://schemas.xmlsoap.org/soap/
envelope/"><ns00:Header></
ns00:Header><ns00:Body><ns01:GetMultipleResourceProperties
xmlns:ns01="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties-1.2-draft-01.xsd
"><ns01:ResourceProperty
xmlns:ns02="http://www.globus.org/namespaces/2004/10/gram/job/
types">ns02:state</ns01:ResourceProperty><ns01:ResourceProperty
xmlns:ns02="http://www.globus.org/namespaces/2004/10/gram/job/
types">ns02:holding</ns01:ResourceProperty><ns01:ResourceProperty
xmlns:ns03="http://www.globus.org/namespaces/2004/10/gram/job/
faults">ns03:fault</ns01:ResourceProperty><ns01:ResourceProperty
xmlns:ns02="http://www.globus.org/namespaces/2004/10/gram/job/types
">ns02:exitCode</ns01:ResourceProperty></
ns01:GetMultipleResourceProperties></ns00:Body></ns00:Envelope>
----------------------------------------------
=== RESPONSE MESSAGE (length 6399) (time 1206627634.546965000) ===
<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";
xmlns:xsd="http://www.w3.org/2001/XMLSchema"; xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance"; xmlns:wsa="
http://schemas.xmlsoap.org/ws/2004/03/
addressing"><soapenv:Header><wsa:MessageID
soapenv:mustUnderstand="0">uuid:f6d14500-fc08-11dc-
a90a-8a2384ad991f</wsa:MessageID><wsa:To
soapenv:mustUnderstand="0">
http://schemas.xmlsoap.org/ws/2004/03/addressing/role/anonymous</
wsa:To><wsa:Action
soapenv:mustUnderstand="0">
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties/GetMultipleResourcePropertiesResponse
</wsa:Action><wsa:From
soapenv:mustUnderstand="0" xmlns:ns4="
http://www.globus.org/namespaces/2004/10/gram/job";><wsa:Address>
https://128.8.120.35:8443/wsrf/services/ManagedExecutableJobService</
wsa:Address><wsa:ReferenceProperties><ns4:ResourceID
xmlns:ns4="http://www.globus.org/namespaces/2004/10/gram/
job">a1253c20-fa72-11dc-a908-8a2384ad991f</ns4:ResourceID></
wsa:ReferenceProperties></wsa:From><wsa:RelatesTo
RelationshipType="wsa:Reply"
soapenv:mustUnderstand="0">uuid:f6d04560-fc08-11dc-
b962-000f1f66888a</wsa:RelatesTo></
soapenv:Header><soapenv:Body><GetMultipleResourcePropertiesResponse
xmlns="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties-1.2-draft-01.xsd
"><ns1:state
xmlns:ns1="http://www.globus.org/namespaces/2004/10/gram/job/
types">Failed</ns1:state><ns2:holding
xmlns:ns2="http://www.globus.org/namespaces/2004/10/gram/job/
types">false</ns2:holding><ns3:fault
xmlns:ns3="http://www.globus.org/namespaces/2004/10/gram/job/
faults"><ns3:invalidPathFault><ns4:Timestamp
xmlns:ns4="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-BaseFaults-1.2-draft-01.xsd
">2008-03-27T06:09:37.203Z</ns4:Timestamp><ns5:Originator xmlns:ns5="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-BaseFaults-1.2-draft-01.xsd
"><wsa:Address>
https://128.8.120.35:8443/wsrf/services/ManagedJobFactoryService</
wsa:Address><wsa:ReferenceProperties><ns6:ResourceID
xmlns:ns6="http://www.globus.org/namespaces/2004/10/gram/
job">a1253c20-fa72-11dc-a908-8a2384ad991f</ns6:ResourceID></
wsa:ReferenceProperties><wsa:ReferenceParameters/></
ns5:Originator><ns7:Description
xmlns:ns7="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-BaseFaults-1.2-draft-01.xsd
">Invalid
executable path
"/export/scratch/applications/a5671f0138bc65dc700001aa80a3f378/
hmmpfam".</ns7:Description><ns8:FaultCause
xmlns:ns8="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-BaseFaults-1.2-draft-01.xsd
"><ns8:Timestamp>2008-03-27T06:09:37.203Z</
ns8:Timestamp><ns8:ErrorCode
dialect="http://www.globus.org/fault/stacktrace";>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(
NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
DelegatingConstructorAccessorImpl.java:27)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:494)
at java.lang.Class.newInstance0(Class.java:350)
at java.lang.Class.newInstance(Class.java:303)
at
org.globus.exec.utils.FaultUtils.makeFault(FaultUtils.java:485)
at org.globus.exec.utils.FaultUtils.createInvalidPathFault(
FaultUtils.java:129)
at
org.globus.exec.service.exec.StateMachine.createFaultFromErrorCode(
StateMachine.java:3184)
at
org
.globus
.exec.service.exec.StateMachine.processWaitingForStateChangesState
(StateMachine.java:1652)
at sun.reflect.GeneratedMethodAccessor6202.invoke(Unknown
Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.globus.exec.service.exec.StateMachine.processState(
StateMachine.java:328)
at org.globus.exec.service.exec.RunThread.run(RunThread.java:85)
</ns8:ErrorCode><ns8:Description>
org.globus.exec.generated.InvalidPathFaultType</ns8:Description></
ns8:FaultCause><ns9:FaultCause
xmlns:ns9="
http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-BaseFaults-1.2-draft-01.xsd
"><ns9:Timestamp>2008-03-27T06:09:37.203Z
</ns9:Timestamp><ns9:Description>ProcessDied</
ns9:Description><ns9:FaultCause><ns9:Timestamp>2008-03-27T06:09:
37.207Z</ns9:Timestamp><ns9:ErrorCode dialect="
http://www.globus.org/fault/stacktrace";>java.lang.Exception:
ProcessDied
at
org.globus.exec.service.exec.StateMachine.createFaultFromErrorCode(
StateMachine.java:3127)
at
org
.globus
.exec.service.exec.StateMachine.processWaitingForStateChangesState
(StateMachine.java:1652)
at sun.reflect.GeneratedMethodAccessor6202.invoke(Unknown
Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.globus.exec.service.exec.StateMachine.processState(
StateMachine.java:328)
at org.globus.exec.service.exec.RunThread.run(RunThread.java:85)
</ns9:ErrorCode><ns9:Description>java.lang.Exception
</ns9:Description></
ns9:FaultCause><ns9:FaultCause><ns9:Timestamp>2008-03-27T06:09:
37.207Z</ns9:Timestamp><ns9:ErrorCode dialect="
http://www.globus.org/fault/stacktrace";>
at
org.globus.wsrf.utils.FaultHelper.toBaseFault(FaultHelper.java
:282)
at
org.globus.exec.utils.FaultUtils.makeFault(FaultUtils.java:505)
at org.globus.exec.utils.FaultUtils.createInvalidPathFault(
FaultUtils.java:129)
at
org.globus.exec.service.exec.StateMachine.createFaultFromErrorCode(
StateMachine.java:3184)
at
org
.globus
.exec.service.exec.StateMachine.processWaitingForStateChangesState
(StateMachine.java:1652)
at sun.reflect.GeneratedMethodAccessor6202.invoke(Unknown
Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.globus.exec.service.exec.StateMachine.processState(
StateMachine.java:328)
at org.globus.exec.service.exec.RunThread.run(RunThread.java:85)
</
ns9:ErrorCode><ns9:Description>org.oasis.wsrf.faults.BaseFaultType</
ns9:Description></ns9:FaultCause></
ns9:FaultCause><ns3:stateWhenFailureOccurred>Active</
ns3:stateWhenFailureOccurred><ns3:command>submit</
ns3:command><ns3:gt2ErrorCode>5</
ns3:gt2ErrorCode><ns3:attribute>executable</ns3:attribute><ns3:path>/
export/scratch/applications/a5671f0138bc65dc700001aa80a3f378/
hmmpfam</ns3:path></ns3:invalidPathFault></ns3:fault><ns10:exitCode
xmlns:ns10="http://www.globus.org/namespaces/2004/10/gram/job/types
">5</ns10:exitCode></GetMultipleResourcePropertiesResponse></
soapenv:Body></soapenv:Envelope>
----------------------------------------------
Current job state: Failed
globusrun-ws: Job failed: Invalid executable path
"/export/scratch/applications/a5671f0138bc65dc700001aa80a3f378/
hmmpfam".
ProcessDied
