On Fri, Aug 22, 2008 at 01:40:00PM -0500, Martin Feller wrote:
> Please try the following:
>
> 1. In the situation when the job hangs:
> How about submitting a job in batch mode (globusrun-ws -submit -b -o
> job.epr ...)
> and query for job status instead of listening for notifications
> (globusrun-ws -status -j job.epr)
> Does the job status change after a while? (I don't expect it, but just to
> make sure)
>
No, still "unsubmitted"
> 2. Shut down the container, enable debug logging in Gram4
> (uncomment # log4j.category.org.globus.exec.service=DEBUG in
> $GLOBUS_LOCATION/container-log4j.properties), clean up the persistence
> directory,
> move the problematic persisted job into the persistence data, start the
> container,
> submit a job.
> Please send the container logfile then.
>
log file attached. I had to increase termination time of that job to
26th, otherwise that file is silently removed and jobs can be submitted
as usual.
Regards,
Yuriy
> Thanks, Martin
>
>
> Yuriy wrote:
>> Hi,
>>
>> I am having very strange problems with globus GRAM.
>>
>>
>> Submission of job with globusrun-ws hangs on "Job Unsubmitted"
>> message. I tried to submit job from two different machines with the
>> same result.
>>
>> globusrun-ws -submit -J -S -F ng2.auckland.ac.nz:8443 -Ft Fork -o
>> test.epr -c /bin/echo "hello"
>> Delegating user credentials...Done.
>> Submitting job...Done.
>> Job ID: uuid:6eeadb2c-6ffa-11dd-a2f7-00163e000005
>> Termination time: 08/23/2008 03:28 GMT
>> Current job state: Unsubmitted
>>
>>
>> Sample java program (attached) and CoG client
>> (cog-job-submit) work normally.
>>
>>
>>
>> Globus restart does not help, unless I remove persisted
>> directory. Persisted is on local partition. I figured that single
>> file in ManagedExecutableJobResourceStateType causes the problem (xml
>> attached). When I remove this file and restart globus, globusws-run
>> works normally. When I copy this file into
>> persisted/ManagedExecutableJobResourceState, and restart globus, it
>> breaks again. My globus breaks every 3-7 days so there are other job
>> resouces that cause this problem.
>>
>> globus version is 4.0.7 from VDT 1.10
>>
>> What is going on here?
>>
>> Regards,
>> Yuriy
>>
>
2008-08-25 10:45:48,478 WARN utils.JavaUtils [main,isAttachmentSupported:1218]
Unable to find required classes (javax.activation.DataHandler and
javax.mail.internet.MimeMultipart). Attachment support is disabled.
2008-08-25 10:45:49,278 DEBUG authorization.GridMapAuthorization
[main,initialize:73] service ManagedJobFactoryService
2008-08-25 10:45:50,778 DEBUG authorization.GridMapAuthorization
[Thread-22,initialize:73] service null
2008-08-25 10:45:50,806 INFO exec.ManagedExecutableJobHome
[Thread-22,recover:207] Recovered resource with ID
91b6ff30-6f57-11dd-a716-ec9d0a188c65.
2008-08-25 10:45:50,867 ERROR delegation.DelegationUtil
[RunQueueThread_0,getDelegationResource:253] Error getting delegation resource
org.globus.wsrf.NoSuchResourceException
at
org.globus.delegation.service.DelegationResource.load(DelegationResource.java:405)
at
org.globus.delegation.service.DelegationHome.find(DelegationHome.java:53)
at
org.globus.delegation.DelegationUtil.getDelegationResource(DelegationUtil.java:251)
at
org.globus.delegation.DelegationUtil.registerDelegationListener(DelegationUtil.java:166)
at
org.globus.exec.service.utils.DelegatedCredential.getDelegatedCredential(DelegatedCredential.java:174)
at
org.globus.exec.service.job.ManagedJobResourceImpl.getJobCredential(ManagedJobResourceImpl.java:421)
at
org.globus.exec.service.exec.StateMachine.processRestartState(StateMachine.java:744)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.globus.exec.service.exec.StateMachine.processState(StateMachine.java:329)
at org.globus.exec.service.exec.RunThread.run(RunThread.java:85)
2008-08-25 10:45:50,895 DEBUG authorization.GridMapAuthorization
[main,initialize:73] service ReliableFileTransferFactoryService
2008-08-25 10:45:51,321 DEBUG authorization.GridMapAuthorization
[main,initialize:73] service ReliableFileTransferService
2008-08-25 10:46:12,961 DEBUG authorization.GridMapAuthorization
[ServiceThread-137,initialize:73] service DelegationFactoryService
2008-08-25 10:46:13,440 DEBUG authorization.GridMapAuthorization
[ServiceThread-138,isPermitted:99] Grid map authz
2008-08-25 10:46:13,440 DEBUG authorization.GridMapAuthorization
[ServiceThread-138,isPermitted:124] Service DelegationFactoryService
2008-08-25 10:46:13,442 DEBUG authorization.GridMapAuthorization
[ServiceThread-138,isPermitted:182] Peer "/C=NZ/O=BeSTGRID/OU=The University of
Auckland/CN=Yuriy Halytskyy" authorized as "grid-bestgrid" based on gridmap
file "/etc/grid-security/grid-mapfile"
2008-08-25 10:46:13,442 INFO authorization.ServiceAuthorizationChain
[ServiceThread-138,authorize:285] Authorized "/C=NZ/O=BeSTGRID/OU=The
University of Auckland/CN=Yuriy Halytskyy" to invoke
"{http://www.globus.org/08/2004/delegationService}requestSecurityToken";.
2008-08-25 10:46:13,454 DEBUG authorization.GridMapAuthorization
[ServiceThread-138,initialize:73] service DelegationService
2008-08-25 10:46:13,461 DEBUG authorization.GridMapAuthorization
[ServiceThread-138,initialize:73] service null
2008-08-25 10:46:13,651 DEBUG authorization.GridMapAuthorization
[ServiceThread-135,isPermitted:99] Grid map authz
2008-08-25 10:46:13,652 DEBUG authorization.GridMapAuthorization
[ServiceThread-135,isPermitted:124] Service ManagedJobFactoryService
2008-08-25 10:46:13,652 DEBUG authorization.GridMapAuthorization
[ServiceThread-135,isPermitted:182] Peer "/C=NZ/O=BeSTGRID/OU=The University of
Auckland/CN=Yuriy Halytskyy" authorized as "grid-bestgrid" based on gridmap
file "/etc/grid-security/grid-mapfile"
2008-08-25 10:46:13,652 INFO authorization.ServiceAuthorizationChain
[ServiceThread-135,authorize:285] Authorized "/C=NZ/O=BeSTGRID/OU=The
University of Auckland/CN=Yuriy Halytskyy" to invoke
"{http://www.globus.org/namespaces/2004/10/gram/job}createManagedJob";.
2008-08-25 10:46:13,668 DEBUG authorization.GridMapAuthorization
[ServiceThread-135,initialize:73] service null
2008-08-25 10:46:20,905 DEBUG authorization.GridMapAuthorization
[ServiceThread-136,initialize:73] service DefaultIndexService
2008-08-25 10:46:20,907 INFO impl.DefaultIndexService
[ServiceThread-136,processConfigFile:107] Reading default registration
configuration from file: /opt/vdt/globus/etc/globus_wsrf_mds_index/hierarchy.xml
2008-08-25 10:46:20,912 INFO impl.DefaultIndexService
[ServiceThread-136,performDefaultRegistrations:193] Processing upstream
registration to https://mds0.arcs.org.au:8443/wsrf/services/DefaultIndexService
2008-08-25 10:46:21,033 INFO impl.DefaultIndexService
[ServiceThread-136,performDefaultRegistrations:193] Processing upstream
registration to https://mds1.arcs.org.au:8443/wsrf/services/DefaultIndexService
2008-08-25 10:46:21,517 DEBUG authorization.GridMapAuthorization
[ServiceThread-136,isPermitted:99] Grid map authz
2008-08-25 10:46:21,517 DEBUG authorization.GridMapAuthorization
[ServiceThread-136,isPermitted:124] Service DefaultIndexService
2008-08-25 10:46:21,518 WARN authorization.GridMapAuthorization
[ServiceThread-136,isPermitted:171] Gridmap authorization failed: peer
"<anonymous>" not in gridmap file "/etc/grid-security/mds-grid-mapfile"
2008-08-25 10:46:21,518 WARN authorization.ServiceAuthorizationChain
[ServiceThread-136,authorize:292] "<anonymous>" is not authorized to use
operation: {http://mds.globus.org/index/2004/07/12}add on this service
2008-08-25 10:46:21,537 WARN client.ServiceGroupRegistrationClient
[Timer-5,status:472] Warning: Could not register
https://130.216.189.3:8443/wsrf/services/ManagedJobFactoryService to
servicegroup at https://130.216.189.3:8443/wsrf/services/DefaultIndexService --
check the URL and that the remote service is up. Remote exception was
org.globus.wsrf.impl.security.authorization.exceptions.AuthorizationException:
"<anonymous>" is not authorized to use operation:
{http://mds.globus.org/index/2004/07/12}add on this service
2008-08-25 10:46:21,638 DEBUG authorization.GridMapAuthorization
[ServiceThread-137,isPermitted:99] Grid map authz
2008-08-25 10:46:21,638 DEBUG authorization.GridMapAuthorization
[ServiceThread-137,isPermitted:124] Service null
2008-08-25 10:46:21,639 WARN authorization.GridMapAuthorization
[ServiceThread-137,isPermitted:171] Gridmap authorization failed: peer
"<anonymous>" not in gridmap file "/etc/grid-security/grid-mapfile"
2008-08-25 10:46:21,639 WARN authorization.ServiceAuthorizationChain
[ServiceThread-137,authorize:292] "<anonymous>" is not authorized to use
operation: {http://mds.globus.org/index/2004/07/12}add on this service
2008-08-25 10:46:21,644 WARN client.ServiceGroupRegistrationClient
[Timer-5,status:472] Warning: Could not register
https://130.216.189.3:8443/wsrf/services/ManagedJobFactoryService to
servicegroup at https://130.216.189.3:8443/wsrf/services/DefaultIndexService --
check the URL and that the remote service is up. Remote exception was
org.globus.wsrf.impl.security.authorization.exceptions.AuthorizationException:
"<anonymous>" is not authorized to use operation:
{http://mds.globus.org/index/2004/07/12}add on this service
2008-08-25 10:46:21,744 DEBUG authorization.GridMapAuthorization
[ServiceThread-138,isPermitted:99] Grid map authz
2008-08-25 10:46:21,744 DEBUG authorization.GridMapAuthorization
[ServiceThread-138,isPermitted:124] Service null
2008-08-25 10:46:21,744 WARN authorization.GridMapAuthorization
[ServiceThread-138,isPermitted:171] Gridmap authorization failed: peer
"<anonymous>" not in gridmap file "/etc/grid-security/grid-mapfile"
2008-08-25 10:46:21,745 WARN authorization.ServiceAuthorizationChain
[ServiceThread-138,authorize:292] "<anonymous>" is not authorized to use
operation: {http://mds.globus.org/index/2004/07/12}add on this service
2008-08-25 10:46:21,749 WARN client.ServiceGroupRegistrationClient
[Timer-5,status:472] Warning: Could not register
https://130.216.189.3:8443/wsrf/services/ManagedJobFactoryService to
servicegroup at https://130.216.189.3:8443/wsrf/services/DefaultIndexService --
check the URL and that the remote service is up. Remote exception was
org.globus.wsrf.impl.security.authorization.exceptions.AuthorizationException:
"<anonymous>" is not authorized to use operation:
{http://mds.globus.org/index/2004/07/12}add on this service
2008-08-25 10:46:21,857 DEBUG authorization.GridMapAuthorization
[ServiceThread-135,isPermitted:99] Grid map authz
2008-08-25 10:46:21,857 DEBUG authorization.GridMapAuthorization
[ServiceThread-135,isPermitted:124] Service null
2008-08-25 10:46:21,858 WARN authorization.GridMapAuthorization
[ServiceThread-135,isPermitted:171] Gridmap authorization failed: peer
"<anonymous>" not in gridmap file "/etc/grid-security/grid-mapfile"
2008-08-25 10:46:21,858 WARN authorization.ServiceAuthorizationChain
[ServiceThread-135,authorize:292] "<anonymous>" is not authorized to use
operation: {http://mds.globus.org/index/2004/07/12}add on this service
2008-08-25 10:46:21,869 WARN client.ServiceGroupRegistrationClient
[Timer-5,status:472] Warning: Could not register
https://130.216.189.3:8443/wsrf/services/ReliableFileTransferFactoryService to
servicegroup at https://130.216.189.3:8443/wsrf/services/DefaultIndexService --
check the URL and that the remote service is up. Remote exception was
org.globus.wsrf.impl.security.authorization.exceptions.AuthorizationException:
"<anonymous>" is not authorized to use operation:
{http://mds.globus.org/index/2004/07/12}add on this service
2008-08-25 10:46:34,847 DEBUG authorization.GridMapAuthorization
[ServiceThread-136,isPermitted:99] Grid map authz
2008-08-25 10:46:34,848 DEBUG authorization.GridMapAuthorization
[ServiceThread-136,isPermitted:124] Service null
2008-08-25 10:46:34,848 DEBUG authorization.GridMapAuthorization
[ServiceThread-136,isPermitted:182] Peer "/C=NZ/O=BeSTGRID/OU=The University of
Auckland/CN=Yuriy Halytskyy" authorized as "grid-bestgrid" based on gridmap
2008-08-25 10:46:34,848 INFO authorization.ServiceAuthorizationChain
[ServiceThread-136,authorize:285] Authorized "/C=NZ/O=BeSTGRID/OU=The
University of Auckland/CN=Yuriy Halytskyy" to invoke
"{http://www.globus.org/namespaces/2004/10/gram/job/exec}getMultipleResourceProperties";.
2008-08-25 10:46:36,296 DEBUG authorization.GridMapAuthorization
[ServiceThread-137,isPermitted:99] Grid map authz
2008-08-25 10:46:36,297 DEBUG authorization.GridMapAuthorization
[ServiceThread-137,isPermitted:124] Service null
2008-08-25 10:46:36,297 DEBUG authorization.GridMapAuthorization
[ServiceThread-137,isPermitted:182] Peer "/C=NZ/O=BeSTGRID/OU=The University of
Auckland/CN=Yuriy Halytskyy" authorized as "grid-bestgrid" based on gridmap
2008-08-25 10:46:36,297 INFO authorization.ServiceAuthorizationChain
[ServiceThread-137,authorize:285] Authorized "/C=NZ/O=BeSTGRID/OU=The
University of Auckland/CN=Yuriy Halytskyy" to invoke
"{http://www.globus.org/namespaces/2004/10/gram/job/exec}getMultipleResourceProperties";.
