RE: Serious, intermittent problem with CF Web Service
So the dev site hits the WS fine and the live site is intermittent - and they share the same physical server? Do they share the same instance of CF? Mark A. Kruger, MCSE, CFG (402) 408-3733 ext 105 www.cfwebtools.com www.coldfusionmuse.com www.necfug.com -Original Message- From: Leon Miller-Out [mailto:l...@singlebrook.com] Sent: Tuesday, March 02, 2010 1:27 PM To: cf-talk Subject: Re: Serious, intermittent problem with CF Web Service Nope. They're on the same physical server and using the same DNS. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331263 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
> I doubt that we're going to upgrade to CF9. The thought of paying $4000 to > Adobe for an upgrade that may or may not fix our > problem (and is sure to come with its own new problems) just turns my stomach. Well, I'd definitely recommend that you try the 30-day trial first! You might also want to involve Adobe support - there are no guarantees for resolution on a mailing list. You might even want to involve your Adobe sales rep, if you're currently on a subscription. Even if you didn't have this issue, you might find CF 9 worth the purchase - it's faster, better, stronger, etc. Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ http://training.figleaf.com/ Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on GSA Schedule, and provides the highest caliber vendor-authorized instruction at our training centers, online, or onsite. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331261 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
Nope. They're on the same physical server and using the same DNS. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331260 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
I doubt that we're going to upgrade to CF9. The thought of paying $4000 to Adobe for an upgrade that may or may not fix our problem (and is sure to come with its own new problems) just turns my stomach. I hate having to upgrade for bug fixes! I'm going to be looking at Bluedragon and Railo. I'm getting tired of proprietary, closed-source software because it makes some bug-fixes virtually impossible. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331259 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: Serious, intermittent problem with CF Web Service
Leon, Are your dev and production systems using different DNS servers? -Mark Mark A. Kruger, MCSE, CFG (402) 408-3733 ext 105 www.cfwebtools.com www.coldfusionmuse.com www.necfug.com -Original Message- From: Leon Miller-Out [mailto:l...@singlebrook.com] Sent: Monday, March 01, 2010 8:51 AM To: cf-talk Subject: Re: Serious, intermittent problem with CF Web Service Mark, thanks for your input. We haven't been able to reproduce the error outside of production, and we can't reproduce it at will anywhere. We make the same SOAP calls to the service for days without trouble, then all of sudden it can't find classes that it could find before. We're still getting the error after updating to the "latest" Sun 1.4.2 JRE, and running with "save class files" turned off. I'm going to review the rest of the thread and see if there's any other suggestions that I can try. Leon >You are going to need to trap the raw soap when this error occurs (I'm out >of other ideas). That means running a test bed and trying to get the error >to trigger. Is the error reproducible like that? ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331208 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
> > We're still getting the error after updating to the "latest" Sun 1.4.2 JRE, > > Did I miss why you couldn't try a much more recent one ? He's running CF 7, I think. I don't think you can use a newer JVM with CF 7 without breaking web services. I'd recommend trying CF 9 if that's a possibility. Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ http://training.figleaf.com/ Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on GSA Schedule, and provides the highest caliber vendor-authorized instruction at our training centers, online, or onsite. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331207 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
On Monday 01 Mar 2010, Leon Miller-Out wrote: > We're still getting the error after updating to the "latest" Sun 1.4.2 JRE, Did I miss why you couldn't try a much more recent one ? -- Helping to advantageously maintain IPOs as part of the IT team of the year 2010, '09 and '08 This email is sent for and on behalf of Halliwells LLP. Halliwells LLP is a limited liability partnership registered in England and Wales under registered number OC307980 whose registered office address is at Halliwells LLP, 3 Hardman Square, Spinningfields, Manchester, M3 3EB. A list of members is available for inspection at the registered office together with a list of those non members who are referred to as partners. We use the word partner to refer to a member of the LLP, or an employee or consultant with equivalent standing and qualifications. Regulated by the Solicitors Regulation Authority. CONFIDENTIALITY This email is intended only for the use of the addressee named above and may be confidential or legally privileged. If you are not the addressee you must not read it and must not use any information contained in nor copy it nor inform any person other than Halliwells LLP or the addressee of its existence or contents. If you have received this email in error please delete it and notify Halliwells LLP IT Department on 0870 365 2500. For more information about Halliwells LLP visit www.halliwells.co ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331206 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
Mark, thanks for your input. We haven't been able to reproduce the error outside of production, and we can't reproduce it at will anywhere. We make the same SOAP calls to the service for days without trouble, then all of sudden it can't find classes that it could find before. We're still getting the error after updating to the "latest" Sun 1.4.2 JRE, and running with "save class files" turned off. I'm going to review the rest of the thread and see if there's any other suggestions that I can try. Leon >You are going to need to trap the raw soap when this error occurs (I'm out >of other ideas). That means running a test bed and trying to get the error >to trigger. Is the error reproducible like that? ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331204 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: Serious, intermittent problem with CF Web Service
You are going to need to trap the raw soap when this error occurs (I'm out of other ideas). That means running a test bed and trying to get the error to trigger. Is the error reproducible like that? -mark Mark A. Kruger, MCSE, CFG (402) 408-3733 ext 105 www.cfwebtools.com www.coldfusionmuse.com www.necfug.com -Original Message- From: Leon Miller-Out [mailto:l...@singlebrook.com] Sent: Tuesday, February 23, 2010 9:07 AM To: cf-talk Subject: Re: Serious, intermittent problem with CF Web Service Mark, We encountered the error running with JIT disabled, so JIT is vindicated. The WSDL shouldn't be changing at all. I haven't tried using a static version, and I'm pretty sure it wouldn't solve the problem fully. As I noted earlier, the error sometimes happens when requesting the WSDL and sometimes when invoking a method in the web service. A static WSDL might solve the former, but almost certainly not the latter. Leon >This is confusing to me. I usually have the opposite problem. The stub gets >cached and I can't get it to recompile. You are indicating it recompiles all >the time - or at least often enough to be a problem that almost has to >be JIT - right? Your item 2 below makes me think the WS itself is involved >in some kind of dynamic WSDL compile... perhaps we need to know more about >it. How often does the WSDL change? > >Like Dave, I think I would try a static WSDL and see if that mitigates the >error. If it doesn't then something more dynamic is going on in the bowels >of the WS. > >-mark > >Mark A. Kruger, MCSE, CFG >(402) 408-3733 ext 105 >www.cfwebtools.com >www.coldfusionmuse.com >www.necfug.com > > >> I think JIT is used all the time, not just after X executions of a certain >piece of code. Anyway, we saw the error running without JIT, >> so it's not the guilty party. The WS does refuse to work after the error >is thrown. Some switch gets flipped and CF/Java >> presumably can't find a class that it could find previously. >> >> This is an internal error that seems to happen at two different times: >> 1. Generating the WSDL for the web service. >> 2. Trying to create an object within a web service function. > >JIT is invoked after the JVM has executed the same code some number of >times, actually, I think. But that's neither here nor there. > >You could try creating static WSDL, and see if that reduces the >problem. Just save the generated WSDL, and give that out to your web >service clients. That's kind of a pain, but could be an interesting >test. > >Dave Watts, CTO, Fig Leaf Software >http://www.figleaf.com/ >http://training.figleaf.com/ > >Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on >GSA Schedule, and provides the highest caliber vendor-authorized >instruction at our training centers, online, or onsite. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331104 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Serious, intermittent problem with CF Web Service
Mark, We encountered the error running with JIT disabled, so JIT is vindicated. The WSDL shouldn't be changing at all. I haven't tried using a static version, and I'm pretty sure it wouldn't solve the problem fully. As I noted earlier, the error sometimes happens when requesting the WSDL and sometimes when invoking a method in the web service. A static WSDL might solve the former, but almost certainly not the latter. Leon >This is confusing to me. I usually have the opposite problem. The stub gets >cached and I can't get it to recompile. You are indicating it recompiles all >the time - or at least often enough to be a problem that almost has to >be JIT - right? Your item 2 below makes me think the WS itself is involved >in some kind of dynamic WSDL compile... perhaps we need to know more about >it. How often does the WSDL change? > >Like Dave, I think I would try a static WSDL and see if that mitigates the >error. If it doesn't then something more dynamic is going on in the bowels >of the WS. > >-mark > >Mark A. Kruger, MCSE, CFG >(402) 408-3733 ext 105 >www.cfwebtools.com >www.coldfusionmuse.com >www.necfug.com > > >> I think JIT is used all the time, not just after X executions of a certain >piece of code. Anyway, we saw the error running without JIT, >> so it's not the guilty party. The WS does refuse to work after the error >is thrown. Some switch gets flipped and CF/Java >> presumably can't find a class that it could find previously. >> >> This is an internal error that seems to happen at two different times: >> 1. Generating the WSDL for the web service. >> 2. Trying to create an object within a web service function. > >JIT is invoked after the JVM has executed the same code some number of >times, actually, I think. But that's neither here nor there. > >You could try creating static WSDL, and see if that reduces the >problem. Just save the generated WSDL, and give that out to your web >service clients. That's kind of a pain, but could be an interesting >test. > >Dave Watts, CTO, Fig Leaf Software >http://www.figleaf.com/ >http://training.figleaf.com/ > >Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on >GSA Schedule, and provides the highest caliber vendor-authorized >instruction at our training centers, online, or onsite. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331059 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
RE: Serious, intermittent problem with CF Web Service
This is confusing to me. I usually have the opposite problem. The stub gets cached and I can't get it to recompile. You are indicating it recompiles all the time - or at least often enough to be a problem that almost has to be JIT - right? Your item 2 below makes me think the WS itself is involved in some kind of dynamic WSDL compile... perhaps we need to know more about it. How often does the WSDL change? Like Dave, I think I would try a static WSDL and see if that mitigates the error. If it doesn't then something more dynamic is going on in the bowels of the WS. -mark Mark A. Kruger, MCSE, CFG (402) 408-3733 ext 105 www.cfwebtools.com www.coldfusionmuse.com www.necfug.com -Original Message- From: Dave Watts [mailto:dwa...@figleaf.com] Sent: Monday, February 22, 2010 1:48 PM To: cf-talk Subject: Re: Serious, intermittent problem with CF Web Service > I think JIT is used all the time, not just after X executions of a certain piece of code. Anyway, we saw the error running without JIT, > so it's not the guilty party. The WS does refuse to work after the error is thrown. Some switch gets flipped and CF/Java > presumably can't find a class that it could find previously. > > This is an internal error that seems to happen at two different times: > 1. Generating the WSDL for the web service. > 2. Trying to create an object within a web service function. JIT is invoked after the JVM has executed the same code some number of times, actually, I think. But that's neither here nor there. You could try creating static WSDL, and see if that reduces the problem. Just save the generated WSDL, and give that out to your web service clients. That's kind of a pain, but could be an interesting test. Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ http://training.figleaf.com/ Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on GSA Schedule, and provides the highest caliber vendor-authorized instruction at our training centers, online, or onsite. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331046 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
> I think JIT is used all the time, not just after X executions of a certain > piece of code. Anyway, we saw the error running without JIT, > so it's not the guilty party. The WS does refuse to work after the error is > thrown. Some switch gets flipped and CF/Java > presumably can't find a class that it could find previously. > > This is an internal error that seems to happen at two different times: > 1. Generating the WSDL for the web service. > 2. Trying to create an object within a web service function. JIT is invoked after the JVM has executed the same code some number of times, actually, I think. But that's neither here nor there. You could try creating static WSDL, and see if that reduces the problem. Just save the generated WSDL, and give that out to your web service clients. That's kind of a pain, but could be an interesting test. Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ http://training.figleaf.com/ Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on GSA Schedule, and provides the highest caliber vendor-authorized instruction at our training centers, online, or onsite. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331043 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Serious, intermittent problem with CF Web Service
>We've been thinking this error happens when the Stub file is regenerated by >JIT right? What about internally with the data that comes back from a >service call ... does it include objects that need defining? In other words >is the error from some additional work done by the WS and NOT from >recompiling the stub? Did you say the WS simply refuse to work after this >error is thrown? > >-Mark I think JIT is used all the time, not just after X executions of a certain piece of code. Anyway, we saw the error running without JIT, so it's not the guilty party. The WS does refuse to work after the error is thrown. Some switch gets flipped and CF/Java presumably can't find a class that it could find previously. This is an internal error that seems to happen at two different times: 1. Generating the WSDL for the web service. 2. Trying to create an object within a web service function. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331040 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
RE: Serious, intermittent problem with CF Web Service
We've been thinking this error happens when the Stub file is regenerated by JIT right? What about internally with the data that comes back from a service call ... does it include objects that need defining? In other words is the error from some additional work done by the WS and NOT from recompiling the stub? Did you say the WS simply refuse to work after this error is thrown? -Mark Mark A. Kruger, MCSE, CFG (402) 408-3733 ext 105 www.cfwebtools.com www.coldfusionmuse.com www.necfug.com -Original Message- From: Leon Miller-Out [mailto:l...@singlebrook.com] Sent: Monday, February 22, 2010 12:23 PM To: cf-talk Subject: Re: Serious, intermittent problem with CF Web Service Thanks to all who've joined this discussion! Here's an update on where things stand at the moment. The service just crapped out for the first time today. One of the cluster instances was still able to generate the WSDL, while the other instance said: AXIS error Sorry, something seems to have gone wrong... here are the details: Exception - java.lang.NoClassDefFoundError: tafkan/remote_api/pfapi/v/trunk/rsp_pf_numeric_array Both cfc-skeletons directories contain a file called tafkan.remote_api.pfapi.v.trunk.rsp_pf_numeric_array.class, and did not appear to contain the otherly-named files we've sometimes seen (remote_api.pfapi.v.trunk.rsp_pf_numeric_array.class). The files in cfc-skeletons do not appear to have been modified since the servers were started yesterday. The uptime on both instances was about 21.5 hours. I was running without JIT (-Xint). I've now restarted both instances. They're now running on Sun Java 1.4.2_19 (instead of _13), and JIT has been re-enabled as it clearly wasn't causing this error and was things were dramatically slower without it. I've also cleared the "save class files" check boxes. And now, we wait again... ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331038 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Serious, intermittent problem with CF Web Service
Thanks to all who've joined this discussion! Here's an update on where things stand at the moment. The service just crapped out for the first time today. One of the cluster instances was still able to generate the WSDL, while the other instance said: AXIS error Sorry, something seems to have gone wrong... here are the details: Exception - java.lang.NoClassDefFoundError: tafkan/remote_api/pfapi/v/trunk/rsp_pf_numeric_array Both cfc-skeletons directories contain a file called tafkan.remote_api.pfapi.v.trunk.rsp_pf_numeric_array.class, and did not appear to contain the otherly-named files we've sometimes seen (remote_api.pfapi.v.trunk.rsp_pf_numeric_array.class). The files in cfc-skeletons do not appear to have been modified since the servers were started yesterday. The uptime on both instances was about 21.5 hours. I was running without JIT (-Xint). I've now restarted both instances. They're now running on Sun Java 1.4.2_19 (instead of _13), and JIT has been re-enabled as it clearly wasn't causing this error and was things were dramatically slower without it. I've also cleared the "save class files" check boxes. And now, we wait again... ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331036 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Serious, intermittent problem with CF Web Service
> What type of load is this web service receiving? Is this > significantly more load than the server had been receiving before > adding the web service? If the CF server load increased because of > adding the web service you may need to retune the JVM settings. And > if you've never tuned the JVM it may be time to do so. > > http://www.trunkful.com/index.cfm/JVM-Tuning > Wil, I'm sure that we could do with a re-tuning of the JVM. It has been years. That said, the load hasn't changed significantly. The server load is usually low (under 0.5 in top), and SeeFusion generally reports that there is less than 1 query/second. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331035 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
> I'll have to check with a colleague about whether we've seen this in a > non-clustered environment. We use clustering more for > stability than for load-balancing. We have gotten Hotspot errors that have > caused CF to crash, and the clustering allows the > service to stay up until we can relaunch the crashed instances. We could > automate this, but it would still result in downtime if we > didn't have the clustering in place. Right. Of course, since you've disabled Hotspot, you won't have that problem! Not that I'm recommending that as a long-term solution. > Done. Do you know if this setting affects cfc-skeletons? Honestly, I'm not sure, but I think it might. I would expect it to. Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ http://training.figleaf.com/ Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on GSA Schedule, and provides the highest caliber vendor-authorized instruction at our training centers, online, or onsite. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331025 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
Thanks for the reply, Dave. See my notes below. > First, can you try this without clustering in place? That is, do you > have a single-server environment you can try? I'll have to check with a colleague about whether we've seen this in a non-clustered environment. We use clustering more for stability than for load-balancing. We have gotten Hotspot errors that have caused CF to crash, and the clustering allows the service to stay up until we can relaunch the crashed instances. We could automate this, but it would still result in downtime if we didn't have the clustering in place. > > Second, can you upgrade your JVM to the latest 1.4.2_xx? You can't > upgrade to 1.5 as it'll break web services, if I recall correctly. Good suggestion. I'll do this with the next CF restart, which will almost undoubtably have to happen today to correct the ClassNotFoundException. > > Third, can you check your environment for Hotspot errors? Jochem > suggested you disable Hotspot, but if you're not getting those errors > that shouldn't be a problem. They look like "hs_err_pid_XXX.log" > somewhere within your CF directory, just search the whole thing to > see > if they're there - I think they get dropped into the same directory > as > the JVM, but am not sure offhand. As I mentioned in response to #1, we do get these, but they're rare, only occurring once every few months or so. When they happen, they take down the whole CF instance. I've disabled the JIT for now to see if it helps with our web services problem. > > Fourth, can you uncheck the "save class files" option in CF > Administrator? Done. Do you know if this setting affects cfc-skeletons? ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331012 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Serious, intermittent problem with CF Web Service
> Does Adobe actually support CF7? We tried to find out and it sounded like > they've outsourced support for it to Fig Leaf Software. Not exactly. We provide access to CF 7 binaries for people who need them, but we're not a formal support channel for CF 7. > If there's anyone who thinks they can get to the bottom of this, I'm pretty > much ready to hire them. We could probably do this for you, but before things go that far, there are a bunch of things you can try. First, can you try this without clustering in place? That is, do you have a single-server environment you can try? Second, can you upgrade your JVM to the latest 1.4.2_xx? You can't upgrade to 1.5 as it'll break web services, if I recall correctly. Third, can you check your environment for Hotspot errors? Jochem suggested you disable Hotspot, but if you're not getting those errors that shouldn't be a problem. They look like "hs_err_pid_XXX.log" somewhere within your CF directory, just search the whole thing to see if they're there - I think they get dropped into the same directory as the JVM, but am not sure offhand. Fourth, can you uncheck the "save class files" option in CF Administrator? Dave Watts, CTO, Fig Leaf Software http://www.figleaf.com/ http://training.figleaf.com/ Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on GSA Schedule, and provides the highest caliber vendor-authorized instruction at our training centers, online, or onsite. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331007 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
Re: Serious, intermittent problem with CF Web Service
If I remember right he said he was running CF 7 which means no built in monitoring and I'm pretty sure that CF 7 does not run on JVM 1.5. Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Feb 21, 2010, at 3:50 PM, Sachin Mohan wrote: > > Leon, > > Couple of things: Why do you say that upgrading to JRE 1.5 is not possible? > I've seen it done seamlessly and very easily in the past and am hoping it > can't be that difficult even with CF web services involved.. > > What does your directory where CF creates the web service stubs look like > after the 24 hours? > > Would you be able to use server monitor that comes inside CFAdmin to see how > many sessions/threads are running over time? Most probably you've already > diagnosed it from that perspective to no avail. > > I guess its difficult diagnosing looking at a black box.. what do the > exception, server logs say when the server quits on you? > > Regards, > Sachin > > > > > ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331006 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
Leon, Couple of things: Why do you say that upgrading to JRE 1.5 is not possible? I've seen it done seamlessly and very easily in the past and am hoping it can't be that difficult even with CF web services involved.. What does your directory where CF creates the web service stubs look like after the 24 hours? Would you be able to use server monitor that comes inside CFAdmin to see how many sessions/threads are running over time? Most probably you've already diagnosed it from that perspective to no avail. I guess its difficult diagnosing looking at a black box.. what do the exception, server logs say when the server quits on you? Regards, Sachin ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331005 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
What type of load is this web service receiving? Is this significantly more load than the server had been receiving before adding the web service? If the CF server load increased because of adding the web service you may need to retune the JVM settings. And if you've never tuned the JVM it may be time to do so. http://www.trunkful.com/index.cfm/JVM-Tuning Wil Genovese One man with courage makes a majority. -Andrew Jackson A fine is a tax for doing wrong. A tax is a fine for doing well. On Feb 21, 2010, at 2:43 PM, Leon Miller-Out wrote: > > Matthew, I'm on Linux, not Windows, so Process Monitor won't help here. I > don't think I'm going be able to figure this out with a Java debugger. That's > just beyond my capabilities. > > Does Adobe actually support CF7? We tried to find out and it sounded like > they've outsourced support for it to Fig Leaf Software. > > If there's anyone who thinks they can get to the bottom of this, I'm pretty > much ready to hire them. > > Leon > >> You can try running Process Monitor (http://technet.microsoft. >> com/en-us/sysinternals/bb896645.aspx) while the application is in the >> bad state. Once you reproduce the error, check the log for the >> location that the app is trying to load the class and ensure it looks >> correct. If you don't see anything amiss, you may have to fire up a >> Java debugger to figure out what might be causing the FNF exception. >> You could also open up a support case with Adobe and work with them to >> figure it out. >> >> >> - Matt Small >> >> > > > ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331004 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
On Sun, Feb 21, 2010 at 9:37 PM, Leon Miller-Out wrote: > There shouldn't be anything in the application that expires. The > applicationTimeout is 1 day, but my understanding is that this just > garbage-collects the application scope if it hasn't been used in a day. Then there should be no problem in setting it to 14 days just to rule it out as a possible cause. Jochem -- Jochem van Dieten http://jochem.vandieten.net/ ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331003 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
Matthew, I'm on Linux, not Windows, so Process Monitor won't help here. I don't think I'm going be able to figure this out with a Java debugger. That's just beyond my capabilities. Does Adobe actually support CF7? We tried to find out and it sounded like they've outsourced support for it to Fig Leaf Software. If there's anyone who thinks they can get to the bottom of this, I'm pretty much ready to hire them. Leon > You can try running Process Monitor (http://technet.microsoft. > com/en-us/sysinternals/bb896645.aspx) while the application is in the > bad state. Once you reproduce the error, check the log for the > location that the app is trying to load the class and ensure it looks > correct. If you don't see anything amiss, you may have to fire up a > Java debugger to figure out what might be causing the FNF exception. > You could also open up a support case with Adobe and work with them to > figure it out. > > > - Matt Small > > ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331002 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
Matthew, I'm on Linux, not Windows, so Process Monitor won't help here. I don't think I'm going be able to figure this out with a Java debugger. That's just beyond my capabilities. Does Adobe actually support CF7? We tried to find out and it sounded like they've outsourced support for it to Fig Leaf Software. If there's anyone who thinks they can get to the bottom of this, I'm pretty much ready to hire them. Leon > You can try running Process Monitor (http://technet.microsoft. > com/en-us/sysinternals/bb896645.aspx) while the application is in the > bad state. Once you reproduce the error, check the log for the > location that the app is trying to load the class and ensure it looks > correct. If you don't see anything amiss, you may have to fire up a > Java debugger to figure out what might be causing the FNF exception. > You could also open up a support case with Adobe and work with them to > figure it out. > > > - Matt Small > > > > Howdy, CFers! We've got an incredibly frustrating situation with a > CF > > Web Services-based API that we wrote and maintain. We had an API in > > > place for years that was stable and working happily with Ruby, PHP, > > > and ColdFusion clients. Then this year a .NET client came along, and > > > we found that our web service was not interoperable with > > statically-typed languages due to our extensive use of structs. > > > > We eventually realized we had to re-write the API without structs, > and > > we've done so. It now uses scaler values, arrays, and CFCs (which > get > > translated to SOAP complexTypes). The .NET client is happy, and we > > wrote proof-of-concept clients in about 6 different languages to > > ensure that we'd be interoperable this time around. > > > > To our great dismay, it appears that our ColdFusion 7 servers can't > > > serve the new API reliably. It works for about a day or so after > > restarting, then the clients start getting errors like: > > > > > Error: coldfusion.xml.rpc.CFCInvocationException > > [java.lang.ClassNotFoundException : tafkan.remote_api.pfapi.v.trunk. > > > rsp_pf_survey_status_array] > > > > and > > > > java.lang.NoClassDefFoundError: > > tafkan/remote_api/pfapi/v/trunk/pf_unit > > > > > > Restarting the CF instances is the only way to make the problem go > > away. A lot of time and money was put into rebuilding the API, so > > everyone is really at wit's end about this. > > > > We've noticed that the WEB-INF/cfc-skeletons directories of our CF > > instances eventually seem to have two copies of the classes for each > > > of the CFCs used by the API. For example: > > > > -rw-r--r-- Feb 17 09:15 remote_api.pfapi.v.trunk.pf_datum.class > > -rw-r--r-- Feb 3 12:20 tafkan.remote_api.pfapi.v.trunk.pf_datum. > > class > > > > It seems like the errors are coming from a namespace or class search > > > path problem, so we tried switching all CFC references to be > > fully-qualified (dot notation starting with a mapping) instead of > just > > simple references to CFCs in the current directory. This seemed > > promising, but the problem came back within 24 hours. > > > > Environment: > > ColdFusion 7,0,2,142559 with hf702-70523, 2-instance cluster > > Sun Java 1.4.2_13 > > Apache 2.0.52 > > Centos 4.5 32-bit > > > > Maybe upgrading one of these venerable pieces of software would > help? > > Maybe upgrading just AXIS? > > > > We need help! I'm sure that there is someone out there with more > > CF/AXIS/SOAP experience than us that can help us get this problem > > resolved. Adobe support doesn't seem to be an option, as CF7 is > EOL'ed > > and in extended-extended support (and that just for a few more days). > > > We will pay the right person good money to help us figure this out. > If > > you're that person, or think you might know who they are, please > > contact me ASAP! > > > > Thanks for reading this mega-post! > > Leon ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331001 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
> On Sat, Feb 20, 2010 at 3:06 AM, Leon Miller-Out wrote: > > To our great dismay, it appears that our ColdFusion 7 servers can't > serve the new API reliably. It works for about a day or so after > restarting, then the clients start getting errors like: > > > > Error: coldfusion.xml.rpc.CFCInvocationException > > [java.lang.ClassNotFoundException : tafkan.remote_api.pfapi.v.trunk. > rsp_pf_survey_status_array] > > The interesting part is that this occurs after some time instead of > immediately. I have not experienced your issue before, bt that > particular behaviour leads me in the following directions: > > 1. Disable the JIT compiler. This compiler recompiles the Java > bytecode into native code after it has been executed a number of > times, so that is obviously something that would happen after a while. > > The downside is that your code will not speed up after X executions. > The way to disable the JIT compiler is to add -Xint to your Java > parameters in jvm.config. See > > 2. Is anything expiring in your application after a while? > Application > variables, framwork reinit or something? > > Jochem > > > -- > Jochem van Dieten > http://jochem.vandieten.net/ Thanks, Jochem! I just tried disabling the JIT compiler. The API test suite now takes twice as long to run. I've read that this change basically trades performance for stability, but I don't know if I can live with every request taking twice as long. We'll see what happens. There shouldn't be anything in the application that expires. The applicationTimeout is 1 day, but my understanding is that this just garbage-collects the application scope if it hasn't been used in a day. I'll see if I can think of anything else. I also tried updating to JRE 1.5, but it's a nightmare with CF7 if you're using Web Services, so I backed it out. ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:331000 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
You can try running Process Monitor (http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx) while the application is in the bad state. Once you reproduce the error, check the log for the location that the app is trying to load the class and ensure it looks correct. If you don't see anything amiss, you may have to fire up a Java debugger to figure out what might be causing the FNF exception. You could also open up a support case with Adobe and work with them to figure it out. - Matt Small > Howdy, CFers! We've got an incredibly frustrating situation with a CF > Web Services-based API that we wrote and maintain. We had an API in > place for years that was stable and working happily with Ruby, PHP, > and ColdFusion clients. Then this year a .NET client came along, and > we found that our web service was not interoperable with > statically-typed languages due to our extensive use of structs. > > We eventually realized we had to re-write the API without structs, and > we've done so. It now uses scaler values, arrays, and CFCs (which get > translated to SOAP complexTypes). The .NET client is happy, and we > wrote proof-of-concept clients in about 6 different languages to > ensure that we'd be interoperable this time around. > > To our great dismay, it appears that our ColdFusion 7 servers can't > serve the new API reliably. It works for about a day or so after > restarting, then the clients start getting errors like: > > Error: coldfusion.xml.rpc.CFCInvocationException > [java.lang.ClassNotFoundException : tafkan.remote_api.pfapi.v.trunk. > rsp_pf_survey_status_array] > > and > > java.lang.NoClassDefFoundError: > tafkan/remote_api/pfapi/v/trunk/pf_unit > > > Restarting the CF instances is the only way to make the problem go > away. A lot of time and money was put into rebuilding the API, so > everyone is really at wit's end about this. > > We've noticed that the WEB-INF/cfc-skeletons directories of our CF > instances eventually seem to have two copies of the classes for each > of the CFCs used by the API. For example: > > -rw-r--r-- Feb 17 09:15 remote_api.pfapi.v.trunk.pf_datum.class > -rw-r--r-- Feb 3 12:20 tafkan.remote_api.pfapi.v.trunk.pf_datum. > class > > It seems like the errors are coming from a namespace or class search > path problem, so we tried switching all CFC references to be > fully-qualified (dot notation starting with a mapping) instead of just > simple references to CFCs in the current directory. This seemed > promising, but the problem came back within 24 hours. > > Environment: > ColdFusion 7,0,2,142559 with hf702-70523, 2-instance cluster > Sun Java 1.4.2_13 > Apache 2.0.52 > Centos 4.5 32-bit > > Maybe upgrading one of these venerable pieces of software would help? > Maybe upgrading just AXIS? > > We need help! I'm sure that there is someone out there with more > CF/AXIS/SOAP experience than us that can help us get this problem > resolved. Adobe support doesn't seem to be an option, as CF7 is EOL'ed > and in extended-extended support (and that just for a few more days). > We will pay the right person good money to help us figure this out. If > you're that person, or think you might know who they are, please > contact me ASAP! > > Thanks for reading this mega-post! > Leon ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:330986 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Re: Serious, intermittent problem with CF Web Service
On Sat, Feb 20, 2010 at 3:06 AM, Leon Miller-Out wrote: > To our great dismay, it appears that our ColdFusion 7 servers can't serve the > new API reliably. It works for about a day or so after restarting, then the > clients start getting errors like: > > Error: coldfusion.xml.rpc.CFCInvocationException > [java.lang.ClassNotFoundException : > tafkan.remote_api.pfapi.v.trunk.rsp_pf_survey_status_array] The interesting part is that this occurs after some time instead of immediately. I have not experienced your issue before, bt that particular behaviour leads me in the following directions: 1. Disable the JIT compiler. This compiler recompiles the Java bytecode into native code after it has been executed a number of times, so that is obviously something that would happen after a while. The downside is that your code will not speed up after X executions. The way to disable the JIT compiler is to add -Xint to your Java parameters in jvm.config. See 2. Is anything expiring in your application after a while? Application variables, framwork reinit or something? Jochem -- Jochem van Dieten http://jochem.vandieten.net/ ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:330985 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
Serious, intermittent problem with CF Web Service
Howdy, CFers! We've got an incredibly frustrating situation with a CF Web Services-based API that we wrote and maintain. We had an API in place for years that was stable and working happily with Ruby, PHP, and ColdFusion clients. Then this year a .NET client came along, and we found that our web service was not interoperable with statically-typed languages due to our extensive use of structs. We eventually realized we had to re-write the API without structs, and we've done so. It now uses scaler values, arrays, and CFCs (which get translated to SOAP complexTypes). The .NET client is happy, and we wrote proof-of-concept clients in about 6 different languages to ensure that we'd be interoperable this time around. To our great dismay, it appears that our ColdFusion 7 servers can't serve the new API reliably. It works for about a day or so after restarting, then the clients start getting errors like: Error: coldfusion.xml.rpc.CFCInvocationException [java.lang.ClassNotFoundException : tafkan.remote_api.pfapi.v.trunk.rsp_pf_survey_status_array] and java.lang.NoClassDefFoundError: tafkan/remote_api/pfapi/v/trunk/pf_unit Restarting the CF instances is the only way to make the problem go away. A lot of time and money was put into rebuilding the API, so everyone is really at wit's end about this. We've noticed that the WEB-INF/cfc-skeletons directories of our CF instances eventually seem to have two copies of the classes for each of the CFCs used by the API. For example: -rw-r--r-- Feb 17 09:15 remote_api.pfapi.v.trunk.pf_datum.class -rw-r--r-- Feb 3 12:20 tafkan.remote_api.pfapi.v.trunk.pf_datum.class It seems like the errors are coming from a namespace or class search path problem, so we tried switching all CFC references to be fully-qualified (dot notation starting with a mapping) instead of just simple references to CFCs in the current directory. This seemed promising, but the problem came back within 24 hours. Environment: ColdFusion 7,0,2,142559 with hf702-70523, 2-instance cluster Sun Java 1.4.2_13 Apache 2.0.52 Centos 4.5 32-bit Maybe upgrading one of these venerable pieces of software would help? Maybe upgrading just AXIS? We need help! I'm sure that there is someone out there with more CF/AXIS/SOAP experience than us that can help us get this problem resolved. Adobe support doesn't seem to be an option, as CF7 is EOL'ed and in extended-extended support (and that just for a few more days). We will pay the right person good money to help us figure this out. If you're that person, or think you might know who they are, please contact me ASAP! Thanks for reading this mega-post! Leon ~| Want to reach the ColdFusion community with something they want? Let them know on the House of Fusion mailing lists Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:330971 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
