On Oct 6, 2007, at 12:03 PM, Chris Chen wrote: > > Rather than doing all the profiling like you (which I would have > liked to but i was too lazy), I just did a thread dump. It appeared > to be occurring within the VFS package. Resin 3.1.2 under Linux > CentOS 4.x was spending FOREVER inside the Jar loading code. > > Anyways, try installing the newest snapshot version (not the stable > 3.1.2 release, which apparently was too stable for my own good) and > see if it solves your problem.
That's true. The new release has major performance fixes to the jar dependency checking. -- Scott > > Thanks, > Chris > > > On Oct 6, 2007, at 4:03 AM, Mattias Jiderhamn wrote: > >> Thanks for all the suggestions. >> >> Serge wrote: >>> I've seen something like this, and we were able to diagnose it on >>> a Unix >>> environment by using a program called truss >>> (http://www.scit.wlv.ac.uk/cgi-bin/mansec?1+truss) which allows >>> you to >>> watch every file that is getting read. I don't know of a Windows >>> equivalent. Most Java profilers I know of do not help you find IO >>> related issues very well, so you might want to look for a lower >>> level OS >>> tool to see what's changing. >>> >>> What we found was that the application was looking for a class >>> that did >>> not exist. Java does not hold a cache entry for "class not >>> found", so >>> it kept opening every jar in the entire classpath to find said >>> class, >>> numerous times for a single http request because of where this >>> library >>> was used. You might want to check all the resin/lib and WEB-INF/ >>> lib and >>> whatever other classes to see if a non-critical class is missing but >>> being searched for over and over. >> I have watched file access using FileMon on Windows. Unfortunately, I >> don't get enough information to see if I have the same problem you >> had. >> But since the inital start of the application is also slow, I lean >> towards there is a generic problem with class loading and/or >> dependency >> checking. >> There seem to occur several depenency checks during a single request, >> which is why a simple page can take 40 seconds. If I increase the >> dependency check intervall, so that there is only one check during >> the >> request monitored by FileMon, it will finish much faster. >> >> >> Scott wrote: >>> Do you see a difference if you remove the resin.dll? (Which would >>> disable the JNI.) >>> >> Not much. >> >>> Also, what is the stack trace for the slow read() call? >> With JNI, 57% of the time for loading a single page (total 78 s) is >> spent in >> com.caucho.vfs.JniStream.read cased by (only!!!) 22 calls from >> com.caucho.vfs.ReadStream.readBuffer >> com.caucho.vfs.ReadStream.waitForRead >> com.caucho.server.port.TcpConnection.run >> ... >> >> >> Without JNI 43% of the time (total 33 s) is spent in >> java.io.File.lastModified caused by 28000 calls (depency checking >> over >> and over) from >> com.caucho.vfs.FilePath.getLastModified >> com.caucho.vfs.Depend.isModified >> ... >> and 41% of the time is spent in >> java.io.File.length called 28000 times from >> com.caucho.vfs.FilePath.getLength >> com.caucho.vfs.Depend.isModified >> ... >> >> >> I created a simple test page, calling File.length and >> File.lastModified >> 100'000 times and measured the time. I then ran the page in another >> webapp, that does not seem to be affected (on the same computer >> though). >> The results were the same (about 0.2 ms / operation - not sure if >> this >> is normal though; I'll have to compare to the office computer next >> week). >> Anyway, this could indicate the problem is not with the I/O calls >> themselves, but the number of I/O calls... (Which makes Serges tip >> more >> interesting.) Again, I will have to profile at the other computer >> next >> week and compare. >> >> >> Tom Hintz wrote: >>> I’ve seen small TCP/IP MTU sizes cause similar behavior. We had VPN >>> software that set the MTU to less than the IP default, which I think >>> should have been 1500(?). Result was packet fragmentation that >>> caused >>> huge delays in the Microsoft TCP implementation. I think the MTU >>> setting, in this case, was in the VPN software. >> I have not run any VPN software other than the built in PPTP >> client, and >> have not changed the MTU of the NICs. (MTU isn't fetched via DHCP >> is it?) >> I thought maybe there was a problem fetching external DTDs when >> reading >> Hibernate mappings or whatever (lots of profiling time spent >> parsing doc >> types and my ISP has had some issues lately). However, inactivating >> the >> NIC altogether makes no difference. >> >> >>> By the way, the CPU usage is peaking all the way through. Around >>> 49% on >>> my dual core system. >>> (And there is even more memory on the computer with the problem >>> than the >>> other one) >>> >>> >>>> -----Original Message----- >>>> From: [EMAIL PROTECTED] >>>> [mailto:[EMAIL PROTECTED] Behalf Of Mattias >>>> Jiderhamn >>>> Sent: Friday 05 October 2007 11:20 >>>> To: Resin >>>> Subject: [Resin-interest] Slooow file reads (really weird!) >>>> >>>> >>>> Hi list. >>>> I have my J2EE webapp on an external hard drive, which I carry >>>> between >>>> my office and my home computer. >>>> On each computer - running Windows XP and Java 1.5 - I have a Resin >>>> (3.0.22) installation and a shortcut to start Resin with the >>>> server root >>>> on the external drive. >>>> This has worked flawlessly for over a year. >>>> >>>> Now suddenly (ok, after returning from vacation), the >>>> application is >>>> immensely slow - on one of the computers! >>>> Starting Resin and the application now takes anywhere from 3 to 5 >>>> minutes on my home computer, compared to the usual 30 or so >>>> seconds. >>>> Loading a simple page can take 40 seconds. It seems that most of >>>> the >>>> time is spent inside the disk access of the dependency checking. >>>> (Turning dependency checking off was much faster, but still >>>> slower than >>>> normal). >>>> I tried to copy the project to the internal drive to see if there >>>> was >>>> some interface hardware issue - no difference. >>>> I profiled the application with JProfiler, and it thinks there is a >>>> hotspot in com.caucho.vsf.JniStream.read(). Why...? >>>> >>>> I am running out of ideas on what could be wrong and how to track >>>> it down. >>>> >>>> Any tips would be much welcome! >>>> >>>> /Mattias >> >> >> >> _______________________________________________ >> resin-interest mailing list >> resin-interest@caucho.com >> http://maillist.caucho.com/mailman/listinfo/resin-interest > > > > _______________________________________________ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest _______________________________________________ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
