OK, I figured it out (sort of). We are submitting these jobs using the Java
GramJob class. Here's the code that matters, where the "job" variable is an
instance of the GramJob class:
// set Job duration and termination
time
long lifetime = 2 * 7 * 24 * 60 * 60 * 1000; // this represents
two
weeks
serviceDuration = new Date(System.currentTimeMillis() +
lifetime);
job.setDuration(serviceDuration);
serviceTerminationDate = new Date(System.currentTimeMillis() +
lifetime);
job.setTerminationTime(serviceTerminationDate);
A while back I had changed the "2" to a "3" (three weeks), but hadn't made
the code live until today (and had forgotten about this change). I have had
really bizarre things happen when a job exceeds it's 2-week limit (stories
for another time), so I was simply trying to increase this to 3 weeks, which
led to that strange error I reported earlier.
Thanks,
Adam
On 11/5/07, Adam Bazinet <[EMAIL PROTECTED]> wrote:
>
> Just to follow up, I have reissued a new host certificate / container cert
> on host A, with no effect. However, I'm sure the globus user certificate
> (and likewise the simpleCA these are coming from) are OK, because I was able
> to use the same globus user on a different host and jobs are working
> fine. Host A is running a 4.1.0 container, for the record.
>
> Thanks,
> Adam
>
> On 11/5/07, Adam Bazinet <[EMAIL PROTECTED] > wrote:
> >
> > Hi All,
> >
> > Our Grid, which has been functioning perfectly well, seems to have
> > broken now all of a sudden in the sense that all GRAM job submissions from
> > host A to host B, C, D, etc (where B C D are independent remote sites) are
> > failing with the following error:
> >
> > 2007-11-05 16:54:38,950 INFO
> > exec.StateMachine[RunQueueThread_0,logJobAccepted:3193] Job
> > b347a4d0-8be9-11dc-a3ac-ba9ad65cba4d accepted for local user 'globus'
> > 2007-11-05 16:54:39,457 ERROR
> > exec.StateMachine[RunQueueThread_0,fileCleanUp:2730] A secondary fault
> > occured while trying
> > to gracefully fail.
> > AxisFault
> > faultCode:
> > {http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd}General<http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd%7DGeneral>
> > faultSubcode:
> > faultString: [Caused by: Conversion failed [Caused by:
> > java.io.IOException: Not a UTC_TIME]]
> > faultActor:
> > faultNode:
> > faultDetail:
> >
> > {http://xml.apache.org/axis/}stackTrace<http://xml.apache.org/axis/%7DstackTrace>:.
> > Caused by Conversion failed. Caused by
> > COM.claymoresystems.cert.CertificateDecodeException :
> > java.io.IOException: Not a UTC_TIME
> > at COM.claymoresystems.cert.X509Cert.&lt;init&gt;(
> > X509Cert.java:209)
> > at org.globus.gsi.ptls.PureTLSUtil.certificateChainToVector(
> > PureTLSUtil.java :226)
> > at org.globus.gsi.proxy.ProxyPathValidator.validate(
> > ProxyPathValidator.java:216)
> > at
> > org.globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.verifyXMLSignature(
> > WSSecurityEngine.java:299)
> > at org.globus
> > .wsrf.impl.security.authentication.wssec.WSSecurityRequestEngine.verifyXMLSignature(
> > WSSecurityRequestEngine.java:97)
> > at org.
> > globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.handleSignatureElement
> > (WSSecurityEngine.java:116)
> > at org.globus
> > .wsrf.impl.security.authentication.wssec.WSSecurityEngine.processSecurityHeader(
> > WSSecurityEngine.java:516)
> > at org.
> > globus.wsrf.impl.security.authentication.wssec.WSSecurityEngine.processSecurityHeader
> > (WSSecurityEngine.java:482)
> > at org.globus
> > .wsrf.impl.security.authentication.wssec.WSSecurityEngine.processSecurityHeader(
> > WSSecurityEngine.java:397)
> > at org.
> > globus.wsrf.impl.security.authentication.wssec.WSSecurityRequestEngine.processSecurityHeader
> > (WSSecurityRequestEngine.java:61)
> > at org.globus
> > .wsrf.impl.security.authentication.wssec.WSSecurityBasicHandler.handleMessage(
> > WSSecurityBasicHandler.java:43)
> > at org.
> > globus.wsrf.impl.security.authentication.wssec.WSSecurityHandler.handleRequest
> > (WSSecurityHandler.java:21)
> > at org.apache.axis.handlers.HandlerChainImpl.handleRequest(
> > HandlerChainImpl.java:105)
> > at org.apache.axis.handlers.JAXRPCHandler.invoke(
> > JAXRPCHandler.java:52)
> > at org.globus.wsrf.handlers.JAXRPCHandler.invoke(
> > JAXRPCHandler.java:26)
> > at org.apache.axis.strategies.InvocationStrategy.visit(
> > InvocationStrategy.java:32)
> > at org.apache.axis.SimpleChain.doVisiting (SimpleChain.java:118)
> > at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
> > at org.apache.axis.server.AxisServer.invoke(AxisServer.java:248)
> > at org.apache.axis.transport.local.LocalSender.invoke(
> > LocalSender.java :141)
> > at org.apache.axis.strategies.InvocationStrategy.visit(
> > InvocationStrategy.java:32)
> > at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
> > at org.apache.axis.SimpleChain.invoke (SimpleChain.java:83)
> > at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
> > at org.apache.axis.client.Call.invokeEngine(Call.java:2727)
> > at org.apache.axis.client.Call.invoke(Call.java :2710)
> > at org.apache.axis.client.Call.invoke(Call.java:2386)
> > at org.apache.axis.client.Call.invoke(Call.java:2309)
> > at org.apache.axis.client.Call.invoke(Call.java:1766)
> > at org.
> > globus.rft.generated.bindings.ReliableFileTransferFactoryPortTypeSOAPBindingStub.createReliableFileTransfer
> > (ReliableFileTransferFactoryPortTypeSOAPBindingStub.java:874)
> > at org.globus
> > .exec.service.exec.utils.StagingHelper.submitStagingRequest(
> > StagingHelper.java:168)
> > at org. globus.exec.service.exec.StateMachine.fileCleanUp (
> > StateMachine.java:2716)
> > at org.globus
> > .exec.service.exec.StateMachine.processFailureFileCleanUpState(
> > StateMachine.java:2091)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:39)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke (Method.java:585)
> > at org.globus.exec.service.exec.StateMachine.processState(
> > StateMachine.java:302)
> > at org.globus.exec.service.exec.RunThread.run( RunThread.java
> > :85)
> >
> > {
> > http://xml.apache.org/axis/}hostname:seil.umd.edu<http://xml.apache.org/axis/%7Dhostname:seil.umd.edu>
> >
> > [Caused by: Conversion failed [Caused by: java.io.IOException: Not a
> > UTC_TIME]]
> > at org.apache.axis.message.SOAPFaultBuilder.createFault(
> > SOAPFaultBuilder.java :221)
> > at org.apache.axis.message.SOAPFaultBuilder.endElement(
> > SOAPFaultBuilder.java:128)
> > at org.apache.axis.encoding.DeserializationContext.endElement(
> > DeserializationContext.java:1087)
> > at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown
> > Source)
> > at
> > org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown
> > Source)
> > at
> > org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
> > Source)
> > at
> > org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
> > Source)
> > at org.apache.xerces.parsers.XML11Configuration.parse(Unknown
> > Source)
> > at org.apache.xerces.parsers.XML11Configuration.parse (Unknown
> > Source)
> > at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
> > at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown
> > Source)
> > at javax.xml.parsers.SAXParser.parse(SAXParser.java :375)
> > at org.apache.axis.encoding.DeserializationContext.parse(
> > DeserializationContext.java:227)
> > at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:645)
> > at org.apache.axis.Message.getSOAPEnvelope (Message.java:424)
> > at
> > org.apache.axis.message.addressing.handler.AddressingHandler.processClientResponse
> > (AddressingHandler.java:305)
> > at
> > org.apache.axis.message.addressing.handler.AddressingHandler.invoke (
> > AddressingHandler.java:110)
> > at org.apache.axis.strategies.InvocationStrategy.visit(
> > InvocationStrategy.java:32)
> > at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
> > at org.apache.axis.SimpleChain.invoke (SimpleChain.java:83)
> > at org.apache.axis.client.AxisClient.invoke(AxisClient.java:190)
> > at org.apache.axis.client.Call.invokeEngine(Call.java:2727)
> > at org.apache.axis.client.Call.invoke(Call.java :2710)
> > at org.apache.axis.client.Call.invoke(Call.java:2386)
> > at org.apache.axis.client.Call.invoke(Call.java:2309)
> > at org.apache.axis.client.Call.invoke(Call.java:1766)
> > at org.
> > globus.rft.generated.bindings.ReliableFileTransferFactoryPortTypeSOAPBindingStub.createReliableFileTransfer
> > (ReliableFileTransferFactoryPortTypeSOAPBindingStub.java:874)
> > at org.globus
> > .exec.service.exec.utils.StagingHelper.submitStagingRequest(
> > StagingHelper.java:168)
> > at org. globus.exec.service.exec.StateMachine.fileCleanUp (
> > StateMachine.java:2716)
> > at org.globus
> > .exec.service.exec.StateMachine.processFailureFileCleanUpState(
> > StateMachine.java:2091)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:39)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke (Method.java:585)
> > at org.globus.exec.service.exec.StateMachine.processState(
> > StateMachine.java:302)
> > at org.globus.exec.service.exec.RunThread.run( RunThread.java
> > :85)
> > 2007-11-05 16:54:40,046 INFO
> > exec.StateMachine[RunQueueThread_0,logJobFailed:3212] Job
> > b347a4d0-8be9-11dc-a3ac-ba9ad65cba4d failed
> >
> > What is this "Not a UTC_TIME" stuff? I've checked and re-checked the
> > certs, they are all valid and I can't think of anything that has changed,
> > except daylight savings! If anyone has ideas, please let me know ASAP!
> >
> > Thanks,
> > Adam
> >
>
>
