So, either this or the canonCache solves this issue for me on my development environment. But, of course, both of them should be avoided on production systems. The chances that they slip from dev to prod by accident may be small (if one is using well known "good" development strategies and methods, like staging, testing Q&A, reviews, and the like) but still possible. And the fact, that it seems to only really matter in JSF dev mode is curious at the least.
Am 28.10.2023 um 08:24 schrieb Vicente Rossello:
Sorry, that was not correct. The allowLinking is at the resources level. So, in the META-INF/context.xml file: <?xml version="1.0" encoding="UTF-8"?> <Context allowCasualMultipartParsing="true" reloadable="false" containerSciFilter="org.apache.tomcat.websocket.server.WsSci|org.apache.jasper.servlet.JasperInitializer" parallelAnnotationScanning="true" clearReferencesThreadLocals="false" skipMemoryLeakChecksOnJvmShutdown="true"> <Manager pathname=""/> <Resources archiveIndexStrategy="bloom" allowLinking="true"/> </Context> BTW, the slowness is something between checking the canonicalPath too many times and jdk12 that removed those canonCache by default. See https://bugs.openjdk.org/browse/JDK-8258036 On Sat, Oct 28, 2023 at 8:07 AM Vicente Rossello <cocorosse...@gmail.com> wrote:Hi, It is a tomcat problem. The real problem is in DirResourcesSet. It is calling a lot of times File.getCanonicalPath(). This method is very slow in windows. What I did is extending the DirResourcesSet to skip processing the getCanonicalPath method so many times, but I really didn't understand the problem. A simpler and better solution is to set allowLinking="true" in the context.xml file. So tomcat does not validate the canonicalPath every single time, which is just too much. Best regards, Vicente On Fri, Oct 27, 2023 at 5:51 PM Jens Zurawski <j...@diegurus.de> wrote:My client is the last Windows System in my little universe. Due to some constraints with my customers I have to deal with it for the time being. The switches, Vicente has provided, solved the problem for me so far. But my "inner monk" is still not satisfied, because practical no one knows of these switches, and after all, TomEE should do a good job on Windows, too. And because 8.0.6 does a good job, and 8.0.8 breaks it to some degree, we should at least try to figure out, what exactly causes this break. Maybe there is an easy way to fix it, then it should be fixed IMHO. So, I will go through the bisect session anyway, as it seems the best possibility to get a grip on it. But I don't want to make these builds on Windows. I'll build them on Linux and will try to ease my life a bit with some scripting. cu Jens Am 27.10.2023 um 17:11 schrieb Richard Zowalla:Great news! (though I am not a Windows user anymore). You should be able to build TomEE on Windows, too. If that does not work out-of-the-box, you could try to build within a docker container or via git-bash in Windows (I am not 100% sure a Windows build will work out of the box) I agree, that it might complicate things for testing purpose as it increases the amount of cycles needed, but that are already very good insights! Gruß Richard Am Freitag, dem 27.10.2023 um 16:55 +0200 schrieb Jens Zurawski:I have a first result on this issue: The problem does not occur if TomEE is running on Linux. I've tested my own built 8.0.8, a freshly extracted 8.0.8 and 8.0.15 from the releases, and none of them showed the dev mode performance problem. Using TomEE 8.0.8 on Windows does show the dev mode performance problem. All of the tested ones, means the released 8.0.8 and my own built 8.0.8 (the one I've built on Linux). My development environment is Windows 10 (Netbeans and therefore also the TomEE for easy deployment and debugging). The problem does occur on stand alone started TomEE and also if started out of Netbeans, so Netbeans can be sorted out as the root of the problem. Java Versions 11, 13, 15 and 17 tested, all the same, so Java should also not be the problem. BTW: It is very unlikely a problem of my individual Windows installation, because this problem also shows up on the Windows system of my colleague, which btw. was the root cause why I'm started to dig deeper into this problem now and why I subscribed to this mailing list ;-) I'm afraid, this will complicate my git bisect session a little bit :-( However, I'll keep you posted. cu Jens Am 27.10.2023 um 13:12 schrieb Jens Zurawski:Thanks Richard (and Vicente), I've changed to JDK 8 and now the builds are successful (both 8.0.6 and 8.0.8). I'll script me some test scripts to ease the build, run and test for every bisect step. A build needs around 3 to 4 minutes. Running a test with my application will need some manual interaction and checks, I guess also something around 4 minutes. This sums up to roughly 8 minutes per step. I've started a bisect and it told me: git bisect good 20441eb git bisect bad 4c8a616 Bisecting: 136 revisions left to test after this (roughly 7 steps) That means, if everything works fine, I should be able to identify the commit after about an hour? (That's cool.... why nobody has told me about git bisect before? ;-) ) Anyway, I will have to find some time for this, maybe I need a couple of days to find some. I will let you know about the outcome. cu Jens Am 27.10.2023 um 11:50 schrieb Richard Zowalla:Hi, you need Java/JDK 8 to build TomEE 8. The Maven command is mvn --fail-at-end clean install -pl tomee/apache-tomee -am - Dfile.encoding=UTF-8 -DskipTests=true You will find the tar.gz/zip in ./tomee/apache-tomee/target Gruß Richard P.S. 8.0.7 is here: https://archive.apache.org/dist/tomee/tomee-8.0.7/ Am Freitag, dem 27.10.2023 um 10:50 +0200 schrieb Jens Zurawski:Hi Richard, to identify the "bad" version was easy. I didn't find a 8.0.7 release on the download page, so I started with 8.0.8. And this version already has the slow dev mode. So for my problem I can say: good: 8.0.6 bad: 8.0.8 Now, I think, I need a little help from you. I tried to build a TomEE out of git (I've cloned the github repository https://github.com/apache/tomee.git). I've tried these two commits: 20441eb (tag: tomee-8.0.6) 4c8a616 (tag: tomee-project-8.0.8) But the build fails every time on "TomEE :: Container :: Core" with: [INFO] BUILD FAILURE [INFO] --------------------------------------------------------------- ------ --- [INFO] Total time: 01:19 min [INFO] Finished at: 2023-10-27T10:29:40+02:00 [INFO] --------------------------------------------------------------- ------ --- [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.2:compile (default-compile) on project openejb-core: Compilation failure: Compilation failure: [ERROR] /sandbox/tomee/container/openejb- core/src/main/java/org/apache/openejb/core/OrbFactory.java:[21, 21] package org.omg.CORBA does not exist [ERROR] /sandbox/tomee/container/openejb- core/src/main/java/org/apache/openejb/core/OrbFactory.java:[24, 12] cannot find symbol [ERROR] symbol: class ORB [ERROR] location: class org.apache.openejb.core.OrbFactory I've tried to build with: mvn clean install My build environment is a linux ubuntu 20.04.6 LTS VM with: git 2.25.1 Maven 3.6.3 java 14.0.2 Is it some easy thing to fix, or do I have to switch some versions of my environment? cu Jens Am 26.10.2023 um 19:48 schrieb Richard Zowalla:Hi, without knowing code or having a reproducer, it might be not that obvious to find a reason / answer for the "academic problem" ;-) If you really want to boil it down, I recommend the following approach: (1) From your description it seems, that 8.0.6 is the last known "good" version. (2) You've investigated versions starting from 8.0.12 onwards, which seems to be "bad". Following the approach recommended by Jonathan: - Can you check if it happens in 8.0.7 (8.0.8, ...) until we find the first "bad" released version? - If we have that, you could start a "git bisect" session between 8.0.6 and the first version, which is known to be bad (reducing the amount of needed bisect cycles dramatically), so we find a commit / reason (if it is in TomEE). - It would involve some quick builds without executing tests (~10- 15min each) for each bisect step and testing on your side. Don't know if your inner monk is willing to go that route, but I am happy to help with the related Maven commands for quick building :) Gruß Richard Am Donnerstag, dem 26.10.2023 um 18:00 +0200 schrieb Jens Zurawski:Hi Vicente, thank you for the tip. This is a good workaround. When I switch to Production mode and insert this configuration, it reloads the view from the file if it has changed. This means, I can use a decent version of TomEE in my dev environment. However, it doesn't completely solve my problem, as the error reporting isn't as verbose as in Dev mode as far as I can see. But I have to check, maybe it's enough for my development. And, well.... the academic problem: "Why is it still broken in Dev mode?" still runs circles through my brain ;-) Because if it's a problem of my coding, it might be a good idea to change this, before it might get a problem on Production mode, too, at some time in the future. But anyway, you've helped me a good step forward, thanks. cu Jens Am 26.10.2023 um 17:04 schrieb Vicente Rossello:We are using the production stage with javax.faces.FACELETS_REFRESH_PERIOD = 1. It's faster and more similar to production, just try it if it solves your problem On Thu, Oct 26, 2023 at 4:18 PM Jens Zurawski <j...@diegurus.de> wrote:The one bundled with TomEE. myfaces 2.3.7 on TomEE 8.0.6 and e.g. myfaces 2.3.9 on TomEE 8.0.12 Am 26.10.2023 um 16:08 schrieb Richard Zowalla:Which JSF lib are you using? Am 26. Oktober 2023 16:00:57 MESZ schrieb Jens Zurawski <j...@diegurus.de :Thank you for your quick reply, Jonathan. It is good to know, that normally the Dev mode should work as fast asbefore even with 8.0.15. So it obviously has something to do with my environment, configuration and/or code.And also thank you for the reference to tomitribe. Unfortunately I'mthe only developer on this project, and it will be difficult to convince my customer to an additional support subscription, because he on production doesn't have issues ;-) But, ok, that's my problem, and maybe I'll consult tomitribe in the future for other projects.So, I'm afraid, for the near future I'm on my own to solve thisproblem. Therefore any hints to restrict the places where I should take a look will be more than welcome.I'm a senior programmer with 20 years of experience in Java programming(SE and JSP). But I'm relative new to JSF/JavaEE/JakartaEE and sometimes I have a hard time to understand which component of the whole stack is responsible for what.The project meanwhile has over a 1/4 million lines of code (most ofthem in the back-end, but also the front-end is big enough to not trace through every lane if I want to get to the point in this life :-D ).The application now runs for nearly 2 years on the customers site andis very reliable and fast in its daily usage. Apart from occasional updates every few month, it will never be restarted or need any manual maintenance at all. So I assume the code is not the badest on earth. At least because of that I'm able to out-rule things like memory leaks, gc issues or bottlenecks (max Heap is never reached, neither half of it, even after running for month).Of course there could be aspects of JSF which I may not have understoodcorrectly until now. But because there are so many new things to a JSF beginner like me, I'm running out of ideas what this could be. It is something that worked well 'til TomEE 8.0.6 (Prod and Dev mode) and suddenly don't work anymore on 8.0.12 in Dev mode (but still works great in Prod mode). There are only minor versions in-between. So I think there were no revolutionary changes anywhere. My hope was (and still is), that anybody here on the list might have had a similar problem and has solved it, or maybe someone knows about some changes in Dev mode which could explain such behaviour (don't know, maybe xml parsing or component tree generation has changed, or scope handling of managed beans, or whatever could cause some code to be entered far more often than before, or some cache/pool was removed or has different behaviour... such things). Then it would be far more easy for me to track down the real problem.Anyway, when I get some spare time (currently have some deadlines insight), I'll try to build a small test case. Either I can get a grip on the issue myself on doing this, or I can supply it to anyone who might want to have a look at it.cu Jens Am 26.10.2023 um 14:01 schrieb Jonathan S. Fisher:newer versions are so incredibly slow in dev modeI can assure you that Dev/Prod mode works very very swiftly on 8.0.15 for sizable applications. For example, I have a giant application (thousands of LOC, 50+ jsf views) that has sub- 8ms response times (minus database time). More than likely they are doing something they should not be doing with JSF apis and are causing problems. I would check the obvious things first with VisualVM: heap size, gc frequency, etc. You can also run their profiler and see if you can spot anything immediately obvious. After that, the easiest way to locate the code slowing you down is bisection. Cut half the code out, run, and continue cutting until you can locate the problematic code. Then after that, isolate a test case where flipping a boolean flag triggers the issue and post the results here. I would get ahold of: https://www.tomitribe.com who are literal experts in this stuff. They offer professional services to handle exactly these situations and can work directly in your codebase to help you find and fix the problem. On Thu, Oct 26, 2023 at 5:48 AM Jens Zurawski <j...@diegurus.de> wrote:Hi altogether, I'm new to this list and hope it's the right place to ask thisquestion.If not, and someone know the right place to ask this, please giveadvise.I'm developing a big JSF Application for a customer. It's a longrunningproject and development and when I started with it, TomEE 8.0.6 wasthemost recent Version of TomEE. The switch to Jakarta is planned fornextyear, so I'm still on the 8.x path for the time being and wasn't abletotest if this problem still exists in the 9.x branch. The problem: On the customers site I'm with the latest TomEE version 8.0.15(runningon Java 11) and everything works fine, because it is running in production mode. But in my development environment I'm still stuckwith8.0.6, because there I need the development mode (javax.faces.PROJECT_STAGE: Development). I need to be able to see changes in facelets without restarting everything everytime, and togetmore detailed error messages. My attempts to update my dev TomEE to something newer than 8.0.6 all failed, because all (at least all I've tested so far) newer versionsareso incredibly slow in dev mode, that it's unbearable to use the application. Several very looong seconds on every request (even the little AJAX requests in a view) is simply not a practical environment. After switching to Production mode everything works very fast, and all my views have response times of very few ms (even the big ones withmax.around 400ms). When switching back to Development mode I have response times of up to 30s on big views. Not with 8.0.6, there even in development mode it's reasonable fast in not getting higher than 1s. My question: What causes this enormous performance degradation? I'm hoping, it is just a configuration which now has another default value or the like.Ifyes, maybe someone can point me in the right direction of where tofindthis configuration? If it's not a configuration thing: what can I dotoget around this? I haven't tested all Versions of TomEE from 8.0.6 to 8.0.15, so Ican'tsay at what version exactly this behaviour changes. If it helps or is needed, I can make some tests to find the exact version where this happens. Versions I've already tested are: 8.0.12, 8.0.13 and 8.0.15. All of them are very slow in dev mode. If you need more information, I'll try to provide it. Thanks in advance for any help cu Jens
