Re: Very slow JSF Development Mode on tomee 8.x newer than 8.0.6
Yes, this may be one of the next steps here. But, I'm afraid, I have not enough info for them right now. Neither do I have a small test case to reproduce the problem, nor can I point them exact enough to what the problem might be. Also, maybe it's only coincidence, and the real problem is somewhere inside myfaces. The fact that ProdMode is not affected at all (at least not enough to measure any difference) and it only arises in JSF DevMode could point in that direction. So, before I go and open an issue on Tomcat, I want to try to narrow down it more. At least to the point where I can say: look, this commit in TomEE introduced it. And even better: here is a small test case to reproduce it. cu Jens Am 28.10.2023 um 12:55 schrieb Richard Zowalla: Cool. Thanks for going down the rabbit hole ;-) Given that there is a lot of information/benchmarks etc. collected in this thread, what do you think about opening an issue in Tomcat with a reference to this thread? They might take a look. Gruß Richard Am 28. Oktober 2023 12:22:12 MESZ schrieb Jens Zurawski : For all who are interested I've made some quick benchmarks, to see the impact. I've taken one of my biggest views which is making a small AJAX poll on the same view every 30s to be able to get some balanced average values for the response times. The payload of the poll is small (around 500 to 1000 bytes) but because it's the same view, the component tree is big. Windows (same machine localhost, it is an 8 core Ryzen-7 with 64GB of RAM and SSD, so the machine is not the limiting factor) ~ 12000 ms DevMode plain ~ 600 ms DevMode with ~ 100 ms ProdMode plain ~ 100 ms ProdMode with Linux (customer site over Internet VPN, it is a 6 core Intel i5 with 16GB of RAM and SSD) ~ 700 ms DevMode plain ~ 350 ms DevMode with ~ 130 ms ProdMode plain ~ 130 ms ProdMode with No, the 12000 ms is _not_ a typo ;-) My conclusions from these figures are: - ProdMode is not affected by this issue - On Windows the problem is very significant and renders the DevMode unusable - On Linux the problem is not that dramatic, so one can say "well, it's dev mode, deal with it", but it is also affected in doubling the response times. Maybe worth enough to also look into this issue even if one is Linux only. cu Jens Am 28.10.2023 um 11:43 schrieb Jens Zurawski: I can confirm, that this setting also solves the problem of the slow dev mode. Thank you for the tip. So, either this or the canonCache solves this issue for me on my development environment. But, of course, both of them should be avoided on production systems. The chances that they slip from dev to prod by accident may be small (if one is using well known "good" development strategies and methods, like staging, testing Q&A, reviews, and the like) but still possible. And the fact, that it seems to only really matter in JSF dev mode is curious at the least. Am 28.10.2023 um 08:24 schrieb Vicente Rossello: Sorry, that was not correct. The allowLinking is at the resources level. So, in the META-INF/context.xml file: BTW, the slowness is something between checking the canonicalPath too many times and jdk12 that removed those canonCache by default. See https://bugs.openjdk.org/browse/JDK-8258036 -- jens zurawski diegurus - zurawski zurawski poppl rohland GbR juister straße 3 65199 wiesbaden kaspersweg 7b 26131 oldenburg internet http://www.diegurus.de tel +49(0)611 72437966 CONFIDENTIALITY NOTICE: This e-mail message is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient but do not wish to receive communications through this medium, please so advise the sender immediately.
Re: Very slow JSF Development Mode on tomee 8.x newer than 8.0.6
Cool. Thanks for going down the rabbit hole ;-) Given that there is a lot of information/benchmarks etc. collected in this thread, what do you think about opening an issue in Tomcat with a reference to this thread? They might take a look. Gruß Richard Am 28. Oktober 2023 12:22:12 MESZ schrieb Jens Zurawski : >For all who are interested I've made some quick benchmarks, to see the impact. >I've taken one of my biggest views which is making a small AJAX poll on the >same view every 30s to be able to get some balanced average values for the >response times. The payload of the poll is small (around 500 to 1000 bytes) >but because it's the same view, the component tree is big. > >Windows (same machine localhost, it is an 8 core Ryzen-7 with 64GB of RAM and >SSD, so the machine is not the limiting factor) > >~ 12000 ms DevMode plain >~ 600 ms DevMode with allowLinking="true"/> >~ 100 ms ProdMode plain >~ 100 ms ProdMode with allowLinking="true"/> > >Linux (customer site over Internet VPN, it is a 6 core Intel i5 with 16GB of >RAM and SSD) > >~ 700 ms DevMode plain >~ 350 ms DevMode with allowLinking="true"/> >~ 130 ms ProdMode plain >~ 130 ms ProdMode with allowLinking="true"/> > >No, the 12000 ms is _not_ a typo ;-) > >My conclusions from these figures are: >- ProdMode is not affected by this issue >- On Windows the problem is very significant and renders the DevMode unusable >- On Linux the problem is not that dramatic, so one can say "well, it's dev >mode, deal with it", but it is also affected in doubling the response times. >Maybe worth enough to also look into this issue even if one is Linux only. > >cu >Jens > > > >Am 28.10.2023 um 11:43 schrieb Jens Zurawski: >> I can confirm, that this setting also solves the problem of the slow dev >> mode. Thank you for the tip. >> >> So, either this or the canonCache solves this issue for me on my development >> environment. But, of course, both of them should be avoided on production >> systems. The chances that they slip from dev to prod by accident may be >> small (if one is using well known "good" development strategies and methods, >> like staging, testing Q&A, reviews, and the like) but still possible. >> And the fact, that it seems to only really matter in JSF dev mode is curious >> at the least. >> >> Am 28.10.2023 um 08:24 schrieb Vicente Rossello: >>> Sorry, that was not correct. >>> >>> The allowLinking is at the resources level. So, in the META-INF/context.xml >>> file: >>> >>> >>> >> containerSciFilter="org.apache.tomcat.websocket.server.WsSci|org.apache.jasper.servlet.JasperInitializer" >>> >>> parallelAnnotationScanning="true" >>> clearReferencesThreadLocals="false" >>> skipMemoryLeakChecksOnJvmShutdown="true"> >>> >>> >>> >>> >>> >>> >>> BTW, the slowness is something between checking the canonicalPath too >>> many times and jdk12 that removed those canonCache by default. See >>> https://bugs.openjdk.org/browse/JDK-8258036 >>> >>>
Re: Very slow JSF Development Mode on tomee 8.x newer than 8.0.6
For all who are interested I've made some quick benchmarks, to see the impact. I've taken one of my biggest views which is making a small AJAX poll on the same view every 30s to be able to get some balanced average values for the response times. The payload of the poll is small (around 500 to 1000 bytes) but because it's the same view, the component tree is big. Windows (same machine localhost, it is an 8 core Ryzen-7 with 64GB of RAM and SSD, so the machine is not the limiting factor) ~ 12000 ms DevMode plain ~ 600 ms DevMode with allowLinking="true"/> ~ 100 ms ProdMode plain ~ 100 ms ProdMode with allowLinking="true"/> Linux (customer site over Internet VPN, it is a 6 core Intel i5 with 16GB of RAM and SSD) ~ 700 ms DevMode plain ~ 350 ms DevMode with allowLinking="true"/> ~ 130 ms ProdMode plain ~ 130 ms ProdMode with allowLinking="true"/> No, the 12000 ms is _not_ a typo ;-) My conclusions from these figures are: - ProdMode is not affected by this issue - On Windows the problem is very significant and renders the DevMode unusable - On Linux the problem is not that dramatic, so one can say "well, it's dev mode, deal with it", but it is also affected in doubling the response times. Maybe worth enough to also look into this issue even if one is Linux only. cu Jens Am 28.10.2023 um 11:43 schrieb Jens Zurawski: I can confirm, that this setting also solves the problem of the slow dev mode. Thank you for the tip. So, either this or the canonCache solves this issue for me on my development environment. But, of course, both of them should be avoided on production systems. The chances that they slip from dev to prod by accident may be small (if one is using well known "good" development strategies and methods, like staging, testing Q&A, reviews, and the like) but still possible. And the fact, that it seems to only really matter in JSF dev mode is curious at the least. Am 28.10.2023 um 08:24 schrieb Vicente Rossello: Sorry, that was not correct. The allowLinking is at the resources level. So, in the META-INF/context.xml file: containerSciFilter="org.apache.tomcat.websocket.server.WsSci|org.apache.jasper.servlet.JasperInitializer" parallelAnnotationScanning="true" clearReferencesThreadLocals="false" skipMemoryLeakChecksOnJvmShutdown="true"> BTW, the slowness is something between checking the canonicalPath too many times and jdk12 that removed those canonCache by default. See https://bugs.openjdk.org/browse/JDK-8258036
Re: Very slow JSF Development Mode on tomee 8.x newer than 8.0.6
I can confirm, that this setting also solves the problem of the slow dev mode. Thank you for the tip. So, either this or the canonCache solves this issue for me on my development environment. But, of course, both of them should be avoided on production systems. The chances that they slip from dev to prod by accident may be small (if one is using well known "good" development strategies and methods, like staging, testing Q&A, reviews, and the like) but still possible. And the fact, that it seems to only really matter in JSF dev mode is curious at the least. Am 28.10.2023 um 08:24 schrieb Vicente Rossello: Sorry, that was not correct. The allowLinking is at the resources level. So, in the META-INF/context.xml file: BTW, the slowness is something between checking the canonicalPath too many times and jdk12 that removed those canonCache by default. See https://bugs.openjdk.org/browse/JDK-8258036 On Sat, Oct 28, 2023 at 8:07 AM Vicente Rossello wrote: Hi, It is a tomcat problem. The real problem is in DirResourcesSet. It is calling a lot of times File.getCanonicalPath(). This method is very slow in windows. What I did is extending the DirResourcesSet to skip processing the getCanonicalPath method so many times, but I really didn't understand the problem. A simpler and better solution is to set allowLinking="true" in the context.xml file. So tomcat does not validate the canonicalPath every single time, which is just too much. Best regards, Vicente On Fri, Oct 27, 2023 at 5:51 PM Jens Zurawski wrote: My client is the last Windows System in my little universe. Due to some constraints with my customers I have to deal with it for the time being. The switches, Vicente has provided, solved the problem for me so far. But my "inner monk" is still not satisfied, because practical no one knows of these switches, and after all, TomEE should do a good job on Windows, too. And because 8.0.6 does a good job, and 8.0.8 breaks it to some degree, we should at least try to figure out, what exactly causes this break. Maybe there is an easy way to fix it, then it should be fixed IMHO. So, I will go through the bisect session anyway, as it seems the best possibility to get a grip on it. But I don't want to make these builds on Windows. I'll build them on Linux and will try to ease my life a bit with some scripting. cu Jens Am 27.10.2023 um 17:11 schrieb Richard Zowalla: Great news! (though I am not a Windows user anymore). You should be able to build TomEE on Windows, too. If that does not work out-of-the-box, you could try to build within a docker container or via git-bash in Windows (I am not 100% sure a Windows build will work out of the box) I agree, that it might complicate things for testing purpose as it increases the amount of cycles needed, but that are already very good insights! Gruß Richard Am Freitag, dem 27.10.2023 um 16:55 +0200 schrieb Jens Zurawski: I have a first result on this issue: The problem does not occur if TomEE is running on Linux. I've tested my own built 8.0.8, a freshly extracted 8.0.8 and 8.0.15 from the releases, and none of them showed the dev mode performance problem. Using TomEE 8.0.8 on Windows does show the dev mode performance problem. All of the tested ones, means the released 8.0.8 and my own built 8.0.8 (the one I've built on Linux). My development environment is Windows 10 (Netbeans and therefore also the TomEE for easy deployment and debugging). The problem does occur on stand alone started TomEE and also if started out of Netbeans, so Netbeans can be sorted out as the root of the problem. Java Versions 11, 13, 15 and 17 tested, all the same, so Java should also not be the problem. BTW: It is very unlikely a problem of my individual Windows installation, because this problem also shows up on the Windows system of my colleague, which btw. was the root cause why I'm started to dig deeper into this problem now and why I subscribed to this mailing list ;-) I'm afraid, this will complicate my git bisect session a little bit :-( However, I'll keep you posted. cu Jens Am 27.10.2023 um 13:12 schrieb Jens Zurawski: Thanks Richard (and Vicente), I've changed to JDK 8 and now the builds are successful (both 8.0.6 and 8.0.8). I'll script me some test scripts to ease the build, run and test for every bisect step. A build needs around 3 to 4 minutes. Running a test with my application will need some manual interaction and checks, I guess also something around 4 minutes. This sums up to roughly 8 minutes per step. I've started a bisect and it told me: git bisect good 20441eb git bisect bad 4c8a616 Bisecting: 136 revisions left to test after this (roughly 7 steps) That means, if everything works fine, I should be able to identify the commit after about an hour? (That's cool why nobody has told me about git bisect before? ;-) ) Anyway, I will have to find some time for this, maybe I need a couple of days to find some. I will let you
Re: Very slow JSF Development Mode on tomee 8.x newer than 8.0.6
Sounds like it should be worth reporting it ;-) Am 28. Oktober 2023 08:51:20 MESZ schrieb Vicente Rossello : >The tomcat regression is probably introduced here: >https://github.com/apache/tomcat/commit/08431bc0b895aa80e78e993a006cabb73aaa6490 > >As it only affects windows and when there are a lot of files (and those >files are constantly refreshed) it's not probably reported by anyone. > > > >On Sat, Oct 28, 2023 at 8:24 AM Vicente Rossello >wrote: > >> Sorry, that was not correct. >> >> The allowLinking is at the resources level. So, in the >> META-INF/context.xml file: >> >> >> > containerSciFilter="org.apache.tomcat.websocket.server.WsSci|org.apache.jasper.servlet.JasperInitializer" >> parallelAnnotationScanning="true" >> clearReferencesThreadLocals="false" >> skipMemoryLeakChecksOnJvmShutdown="true"> >> >> >> >> >> >> >> BTW, the slowness is something between checking the canonicalPath too many >> times and jdk12 that removed those canonCache by default. See >> https://bugs.openjdk.org/browse/JDK-8258036 >> >> >> On Sat, Oct 28, 2023 at 8:07 AM Vicente Rossello >> wrote: >> >>> Hi, >>> >>> It is a tomcat problem. The real problem is in DirResourcesSet. It is >>> calling a lot of times File.getCanonicalPath(). This method is very slow in >>> windows. >>> >>> What I did is extending the DirResourcesSet to skip processing the >>> getCanonicalPath method so many times, but I really didn't understand the >>> problem. >>> >>> A simpler and better solution is to set allowLinking="true" in the >>> context.xml file. So tomcat does not validate the canonicalPath every >>> single time, which is just too much. >>> >>> Best regards, >>> Vicente >>> >>> On Fri, Oct 27, 2023 at 5:51 PM Jens Zurawski wrote: >>> My client is the last Windows System in my little universe. Due to some constraints with my customers I have to deal with it for the time being. The switches, Vicente has provided, solved the problem for me so far. But my "inner monk" is still not satisfied, because practical no one knows of these switches, and after all, TomEE should do a good job on Windows, too. And because 8.0.6 does a good job, and 8.0.8 breaks it to some degree, we should at least try to figure out, what exactly causes this break. Maybe there is an easy way to fix it, then it should be fixed IMHO. So, I will go through the bisect session anyway, as it seems the best possibility to get a grip on it. But I don't want to make these builds on Windows. I'll build them on Linux and will try to ease my life a bit with some scripting. cu Jens Am 27.10.2023 um 17:11 schrieb Richard Zowalla: > Great news! (though I am not a Windows user anymore). > > You should be able to build TomEE on Windows, too. If that does not > work out-of-the-box, you could try to build within a docker container > or via git-bash in Windows > > (I am not 100% sure a Windows build will work out of the box) > > I agree, that it might complicate things for testing purpose as it > increases the amount of cycles needed, but that are already very good > insights! > > Gruß > Richard > > > Am Freitag, dem 27.10.2023 um 16:55 +0200 schrieb Jens Zurawski: >> I have a first result on this issue: >> >> The problem does not occur if TomEE is running on Linux. I've tested >> my >> own built 8.0.8, a freshly extracted 8.0.8 and 8.0.15 from the >> releases, >> and none of them showed the dev mode performance problem. >> >> Using TomEE 8.0.8 on Windows does show the dev mode performance >> problem. >> All of the tested ones, means the released 8.0.8 and my own built >> 8.0.8 >> (the one I've built on Linux). >> >> My development environment is Windows 10 (Netbeans and therefore also >> the TomEE for easy deployment and debugging). The problem does occur >> on >> stand alone started TomEE and also if started out of Netbeans, so >> Netbeans can be sorted out as the root of the problem. Java Versions >> 11, >> 13, 15 and 17 tested, all the same, so Java should also not be the >> problem. BTW: It is very unlikely a problem of my individual Windows >> installation, because this problem also shows up on the Windows >> system >> of my colleague, which btw. was the root cause why I'm started to dig >> deeper into this problem now and why I subscribed to this mailing >> list ;-) >> >> I'm afraid, this will complicate my git bisect session a little bit >> :-( >> >> However, I'll keep you posted. >> >> cu >> Jens >> >> >> Am 27.10.2023 um 13:12 schrieb Jens Zurawski: >>> Thanks Richard (and Vicente), >>> >>> I've changed to JDK 8 and now the builds are successful (both 8.0.6
