Just when I think I've started to learn everything about the JVM, someone posts some strange esoteric JVM switch... I've never even seen those before. More details about what they do here: https://bugs.openjdk.org/browse/JDK-8301001
I also don't know jack about crap about Windows, but if someone said they were having this problem on Linux/OSX, I would say they need to monitor class compilations and file descriptor limits. VisualVM has the ability to monitor class compilations, I would take a peek at that too. On Fri, Oct 27, 2023 at 10:12 AM Richard Zowalla <rich...@zowalla.com> wrote: > > Great news! (though I am not a Windows user anymore). > > You should be able to build TomEE on Windows, too. If that does not > work out-of-the-box, you could try to build within a docker container > or via git-bash in Windows > > (I am not 100% sure a Windows build will work out of the box) > > I agree, that it might complicate things for testing purpose as it > increases the amount of cycles needed, but that are already very good > insights! > > Gruß > Richard > > > Am Freitag, dem 27.10.2023 um 16:55 +0200 schrieb Jens Zurawski: > > I have a first result on this issue: > > > > The problem does not occur if TomEE is running on Linux. I've tested > > my > > own built 8.0.8, a freshly extracted 8.0.8 and 8.0.15 from the > > releases, > > and none of them showed the dev mode performance problem. > > > > Using TomEE 8.0.8 on Windows does show the dev mode performance > > problem. > > All of the tested ones, means the released 8.0.8 and my own built > > 8.0.8 > > (the one I've built on Linux). > > > > My development environment is Windows 10 (Netbeans and therefore also > > the TomEE for easy deployment and debugging). The problem does occur > > on > > stand alone started TomEE and also if started out of Netbeans, so > > Netbeans can be sorted out as the root of the problem. Java Versions > > 11, > > 13, 15 and 17 tested, all the same, so Java should also not be the > > problem. BTW: It is very unlikely a problem of my individual Windows > > installation, because this problem also shows up on the Windows > > system > > of my colleague, which btw. was the root cause why I'm started to dig > > deeper into this problem now and why I subscribed to this mailing > > list ;-) > > > > I'm afraid, this will complicate my git bisect session a little bit > > :-( > > > > However, I'll keep you posted. > > > > cu > > Jens > > > > > > Am 27.10.2023 um 13:12 schrieb Jens Zurawski: > > > Thanks Richard (and Vicente), > > > > > > I've changed to JDK 8 and now the builds are successful (both 8.0.6 > > > and 8.0.8). > > > > > > I'll script me some test scripts to ease the build, run and test > > > for > > > every bisect step. > > > A build needs around 3 to 4 minutes. Running a test with my > > > application will need some manual interaction and checks, I guess > > > also > > > something around 4 minutes. This sums up to roughly 8 minutes per > > > step. > > > > > > I've started a bisect and it told me: > > > git bisect good 20441eb > > > git bisect bad 4c8a616 > > > Bisecting: 136 revisions left to test after this (roughly 7 steps) > > > > > > That means, if everything works fine, I should be able to identify > > > the > > > commit after about an hour? (That's cool.... why nobody has told me > > > about git bisect before? ;-) ) > > > > > > Anyway, I will have to find some time for this, maybe I need a > > > couple > > > of days to find some. > > > I will let you know about the outcome. > > > > > > cu > > > Jens > > > > > > > > > Am 27.10.2023 um 11:50 schrieb Richard Zowalla: > > > > Hi, > > > > > > > > you need Java/JDK 8 to build TomEE 8. > > > > > > > > The Maven command is > > > > > > > > mvn --fail-at-end clean install -pl tomee/apache-tomee -am - > > > > Dfile.encoding=UTF-8 -DskipTests=true > > > > > > > > You will find the tar.gz/zip in ./tomee/apache-tomee/target > > > > > > > > Gruß > > > > Richard > > > > > > > > P.S. 8.0.7 is here: > > > > https://archive.apache.org/dist/tomee/tomee-8.0.7/ > > > > > > > > Am Freitag, dem 27.10.2023 um 10:50 +0200 schrieb Jens Zurawski: > > > > > Hi Richard, > > > > > > > > > > to identify the "bad" version was easy. I didn't find a 8.0.7 > > > > > release > > > > > on > > > > > the download page, so I started with 8.0.8. And this version > > > > > already > > > > > has > > > > > the slow dev mode. So for my problem I can say: > > > > > good: 8.0.6 > > > > > bad: 8.0.8 > > > > > > > > > > Now, I think, I need a little help from you. I tried to build a > > > > > TomEE > > > > > out of git (I've cloned the github repository > > > > > https://github.com/apache/tomee.git). I've tried these two > > > > > commits: > > > > > 20441eb (tag: tomee-8.0.6) > > > > > 4c8a616 (tag: tomee-project-8.0.8) > > > > > > > > > > But the build fails every time on "TomEE :: Container :: Core" > > > > > with: > > > > > > > > > > [INFO] BUILD FAILURE > > > > > [INFO] > > > > > --------------------------------------------------------------- > > > > > ------ > > > > > --- > > > > > [INFO] Total time: 01:19 min > > > > > [INFO] Finished at: 2023-10-27T10:29:40+02:00 > > > > > [INFO] > > > > > --------------------------------------------------------------- > > > > > ------ > > > > > --- > > > > > [ERROR] Failed to execute goal > > > > > org.apache.maven.plugins:maven-compiler-plugin:3.6.2:compile > > > > > (default-compile) on project openejb-core: Compilation failure: > > > > > Compilation failure: > > > > > [ERROR] > > > > > /sandbox/tomee/container/openejb- > > > > > core/src/main/java/org/apache/openejb/core/OrbFactory.java:[21, > > > > > 21] > > > > > package org.omg.CORBA does not exist > > > > > [ERROR] > > > > > /sandbox/tomee/container/openejb- > > > > > core/src/main/java/org/apache/openejb/core/OrbFactory.java:[24, > > > > > 12] > > > > > cannot find symbol > > > > > [ERROR] symbol: class ORB > > > > > [ERROR] location: class org.apache.openejb.core.OrbFactory > > > > > > > > > > I've tried to build with: > > > > > mvn clean install > > > > > > > > > > My build environment is a linux ubuntu 20.04.6 LTS VM with: > > > > > git 2.25.1 > > > > > Maven 3.6.3 > > > > > java 14.0.2 > > > > > > > > > > Is it some easy thing to fix, or do I have to switch some > > > > > versions of > > > > > my > > > > > environment? > > > > > > > > > > cu > > > > > Jens > > > > > > > > > > Am 26.10.2023 um 19:48 schrieb Richard Zowalla: > > > > > > Hi, > > > > > > > > > > > > without knowing code or having a reproducer, it might be not > > > > > > that > > > > > > obvious to find a reason / answer for the "academic problem" > > > > > > ;-) > > > > > > > > > > > > If you really want to boil it down, I recommend the following > > > > > > approach: > > > > > > > > > > > > (1) From your description it seems, that 8.0.6 is the last > > > > > > known > > > > > > "good" > > > > > > version. > > > > > > > > > > > > (2) You've investigated versions starting from 8.0.12 > > > > > > onwards, > > > > > > which > > > > > > seems to be "bad". > > > > > > > > > > > > Following the approach recommended by Jonathan: > > > > > > > > > > > > - Can you check if it happens in 8.0.7 (8.0.8, ...) until we > > > > > > find > > > > > > the > > > > > > first "bad" released version? > > > > > > > > > > > > - If we have that, you could start a "git bisect" session > > > > > > between > > > > > > 8.0.6 > > > > > > and the first version, which is known to be bad (reducing the > > > > > > amount of > > > > > > needed bisect cycles dramatically), so we find a commit / > > > > > > reason > > > > > > (if it > > > > > > is in TomEE). > > > > > > > > > > > > - It would involve some quick builds without executing tests > > > > > > (~10- > > > > > > 15min > > > > > > each) for each bisect step and testing on your side. > > > > > > > > > > > > Don't know if your inner monk is willing to go that route, > > > > > > but I am > > > > > > happy to help with the related Maven commands for quick > > > > > > building :) > > > > > > > > > > > > Gruß > > > > > > Richard > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am Donnerstag, dem 26.10.2023 um 18:00 +0200 schrieb Jens > > > > > > Zurawski: > > > > > > > Hi Vicente, > > > > > > > > > > > > > > thank you for the tip. This is a good workaround. When I > > > > > > > switch > > > > > > > to > > > > > > > Production mode and insert this configuration, it reloads > > > > > > > the > > > > > > > view > > > > > > > from > > > > > > > the file if it has changed. This means, I can use a decent > > > > > > > version of > > > > > > > TomEE in my dev environment. > > > > > > > > > > > > > > However, it doesn't completely solve my problem, as the > > > > > > > error > > > > > > > reporting > > > > > > > isn't as verbose as in Dev mode as far as I can see. But I > > > > > > > have > > > > > > > to > > > > > > > check, maybe it's enough for my development. > > > > > > > > > > > > > > And, well.... the academic problem: "Why is it still broken > > > > > > > in > > > > > > > Dev > > > > > > > mode?" still runs circles through my brain ;-) Because if > > > > > > > it's a > > > > > > > problem > > > > > > > of my coding, it might be a good idea to change this, > > > > > > > before it > > > > > > > might > > > > > > > get a problem on Production mode, too, at some time in the > > > > > > > future. > > > > > > > > > > > > > > But anyway, you've helped me a good step forward, thanks. > > > > > > > cu > > > > > > > Jens > > > > > > > > > > > > > > > > > > > > > Am 26.10.2023 um 17:04 schrieb Vicente Rossello: > > > > > > > > We are using the production stage with > > > > > > > > javax.faces.FACELETS_REFRESH_PERIOD > > > > > > > > = 1. It's faster and more similar to production, just try > > > > > > > > it if > > > > > > > > it > > > > > > > > solves > > > > > > > > your problem > > > > > > > > > > > > > > > > On Thu, Oct 26, 2023 at 4:18 PM Jens Zurawski > > > > > > > > <j...@diegurus.de> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > The one bundled with TomEE. > > > > > > > > > myfaces 2.3.7 on TomEE 8.0.6 > > > > > > > > > and e.g. myfaces 2.3.9 on TomEE 8.0.12 > > > > > > > > > > > > > > > > > > Am 26.10.2023 um 16:08 schrieb Richard Zowalla: > > > > > > > > > > Which JSF lib are you using? > > > > > > > > > > > > > > > > > > > > Am 26. Oktober 2023 16:00:57 MESZ schrieb Jens > > > > > > > > > > Zurawski > > > > > > > > > > <j...@diegurus.de > > > > > > > > > > : > > > > > > > > > > > Thank you for your quick reply, Jonathan. > > > > > > > > > > > > > > > > > > > > > > It is good to know, that normally the Dev mode > > > > > > > > > > > should > > > > > > > > > > > work as > > > > > > > > > > > fast as > > > > > > > > > before even with 8.0.15. So it obviously has something > > > > > > > > > to do > > > > > > > > > with > > > > > > > > > my > > > > > > > > > environment, configuration and/or code. > > > > > > > > > > > And also thank you for the reference to tomitribe. > > > > > > > > > > > Unfortunately I'm > > > > > > > > > the only developer on this project, and it will be > > > > > > > > > difficult > > > > > > > > > to > > > > > > > > > convince my > > > > > > > > > customer to an additional support subscription, because > > > > > > > > > he on > > > > > > > > > production > > > > > > > > > doesn't have issues ;-) But, ok, that's my problem, and > > > > > > > > > maybe > > > > > > > > > I'll consult > > > > > > > > > tomitribe in the future for other projects. > > > > > > > > > > > So, I'm afraid, for the near future I'm on my own > > > > > > > > > > > to > > > > > > > > > > > solve > > > > > > > > > > > this > > > > > > > > > problem. Therefore any hints to restrict the places > > > > > > > > > where I > > > > > > > > > should take a > > > > > > > > > look will be more than welcome. > > > > > > > > > > > I'm a senior programmer with 20 years of experience > > > > > > > > > > > in > > > > > > > > > > > Java > > > > > > > > > > > programming > > > > > > > > > (SE and JSP). But I'm relative new to > > > > > > > > > JSF/JavaEE/JakartaEE > > > > > > > > > and > > > > > > > > > sometimes I > > > > > > > > > have a hard time to understand which component of the > > > > > > > > > whole > > > > > > > > > stack > > > > > > > > > is > > > > > > > > > responsible for what. > > > > > > > > > > > The project meanwhile has over a 1/4 million lines > > > > > > > > > > > of > > > > > > > > > > > code > > > > > > > > > > > (most of > > > > > > > > > them in the back-end, but also the front-end is big > > > > > > > > > enough to > > > > > > > > > not > > > > > > > > > trace > > > > > > > > > through every lane if I want to get to the point in > > > > > > > > > this life > > > > > > > > > :-D > > > > > > > > > ). > > > > > > > > > > > The application now runs for nearly 2 years on the > > > > > > > > > > > customers > > > > > > > > > > > site and > > > > > > > > > is very reliable and fast in its daily usage. Apart > > > > > > > > > from > > > > > > > > > occasional updates > > > > > > > > > every few month, it will never be restarted or need any > > > > > > > > > manual > > > > > > > > > maintenance > > > > > > > > > at all. So I assume the code is not the badest on > > > > > > > > > earth. At > > > > > > > > > least > > > > > > > > > because > > > > > > > > > of that I'm able to out-rule things like memory leaks, > > > > > > > > > gc > > > > > > > > > issues > > > > > > > > > or > > > > > > > > > bottlenecks (max Heap is never reached, neither half of > > > > > > > > > it, > > > > > > > > > even > > > > > > > > > after > > > > > > > > > running for month). > > > > > > > > > > > Of course there could be aspects of JSF which I may > > > > > > > > > > > not > > > > > > > > > > > have > > > > > > > > > > > understood > > > > > > > > > correctly until now. But because there are so many new > > > > > > > > > things > > > > > > > > > to > > > > > > > > > a JSF > > > > > > > > > beginner like me, I'm running out of ideas what this > > > > > > > > > could > > > > > > > > > be. It > > > > > > > > > is > > > > > > > > > something that worked well 'til TomEE 8.0.6 (Prod and > > > > > > > > > Dev > > > > > > > > > mode) > > > > > > > > > and > > > > > > > > > suddenly don't work anymore on 8.0.12 in Dev mode (but > > > > > > > > > still > > > > > > > > > works great in > > > > > > > > > Prod mode). There are only minor versions in-between. > > > > > > > > > So I > > > > > > > > > think > > > > > > > > > there were > > > > > > > > > no revolutionary changes anywhere. My hope was (and > > > > > > > > > still > > > > > > > > > is), > > > > > > > > > that anybody > > > > > > > > > here on the list might have had a similar problem and > > > > > > > > > has > > > > > > > > > solved > > > > > > > > > it, or > > > > > > > > > maybe someone knows about some changes in Dev mode > > > > > > > > > which > > > > > > > > > could > > > > > > > > > explain such > > > > > > > > > behaviour (don't know, maybe xml parsing or component > > > > > > > > > tree > > > > > > > > > generation has > > > > > > > > > changed, or scope handling of managed beans, or > > > > > > > > > whatever > > > > > > > > > could > > > > > > > > > cause some > > > > > > > > > code to be entered far more often than before, or some > > > > > > > > > cache/pool > > > > > > > > > was > > > > > > > > > removed or has different behaviour... such things). > > > > > > > > > Then it > > > > > > > > > would > > > > > > > > > be far > > > > > > > > > more easy for me to track down the real problem. > > > > > > > > > > > Anyway, when I get some spare time (currently have > > > > > > > > > > > some > > > > > > > > > > > deadlines in > > > > > > > > > sight), I'll try to build a small test case. Either I > > > > > > > > > can get > > > > > > > > > a > > > > > > > > > grip on the > > > > > > > > > issue myself on doing this, or I can supply it to > > > > > > > > > anyone who > > > > > > > > > might want to > > > > > > > > > have a look at it. > > > > > > > > > > > cu > > > > > > > > > > > Jens > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Am 26.10.2023 um 14:01 schrieb Jonathan S. Fisher: > > > > > > > > > > > > > newer versions are so incredibly slow in dev > > > > > > > > > > > > > mode > > > > > > > > > > > > I can assure you that Dev/Prod mode works very > > > > > > > > > > > > very > > > > > > > > > > > > swiftly > > > > > > > > > > > > on 8.0.15 > > > > > > > > > > > > for sizable applications. For example, I have a > > > > > > > > > > > > giant > > > > > > > > > > > > application > > > > > > > > > > > > (thousands of LOC, 50+ jsf views) that has sub- > > > > > > > > > > > > 8ms > > > > > > > > > > > > response > > > > > > > > > > > > times > > > > > > > > > > > > (minus database time). More than likely they are > > > > > > > > > > > > doing > > > > > > > > > > > > something they > > > > > > > > > > > > should not be doing with JSF apis and are causing > > > > > > > > > > > > problems. > > > > > > > > > > > > > > > > > > > > > > > > I would check the obvious things first with > > > > > > > > > > > > VisualVM: > > > > > > > > > > > > heap > > > > > > > > > > > > size, gc > > > > > > > > > > > > frequency, etc. You can also run their profiler > > > > > > > > > > > > and see > > > > > > > > > > > > if > > > > > > > > > > > > you can > > > > > > > > > > > > spot anything immediately obvious. > > > > > > > > > > > > > > > > > > > > > > > > After that, the easiest way to locate the code > > > > > > > > > > > > slowing > > > > > > > > > > > > you > > > > > > > > > > > > down is > > > > > > > > > > > > bisection. Cut half the code out, run, and > > > > > > > > > > > > continue > > > > > > > > > > > > cutting > > > > > > > > > > > > until you > > > > > > > > > > > > can locate the problematic code. Then after that, > > > > > > > > > > > > isolate a > > > > > > > > > > > > test case > > > > > > > > > > > > where flipping a boolean flag triggers the issue > > > > > > > > > > > > and > > > > > > > > > > > > post > > > > > > > > > > > > the results > > > > > > > > > > > > here. > > > > > > > > > > > > > > > > > > > > > > > > I would get ahold of: > > > > > > > > > > > > https://www.tomitribe.com who are > > > > > > > > > > > > literal > > > > > > > > > > > > experts in this stuff. They offer professional > > > > > > > > > > > > services > > > > > > > > > > > > to > > > > > > > > > > > > handle > > > > > > > > > > > > exactly these situations and can work directly in > > > > > > > > > > > > your > > > > > > > > > > > > codebase to > > > > > > > > > > > > help you find and fix the problem. > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Oct 26, 2023 at 5:48 AM Jens Zurawski > > > > > > > > > > > > <j...@diegurus.de> wrote: > > > > > > > > > > > > > Hi altogether, > > > > > > > > > > > > > > > > > > > > > > > > > > I'm new to this list and hope it's the right > > > > > > > > > > > > > place to > > > > > > > > > > > > > ask > > > > > > > > > > > > > this > > > > > > > > > question. > > > > > > > > > > > > > If not, and someone know the right place to ask > > > > > > > > > > > > > this, > > > > > > > > > > > > > please give > > > > > > > > > advise. > > > > > > > > > > > > > I'm developing a big JSF Application for a > > > > > > > > > > > > > customer. > > > > > > > > > > > > > It's > > > > > > > > > > > > > a long > > > > > > > > > running > > > > > > > > > > > > > project and development and when I started with > > > > > > > > > > > > > it, > > > > > > > > > > > > > TomEE > > > > > > > > > > > > > 8.0.6 was > > > > > > > > > the > > > > > > > > > > > > > most recent Version of TomEE. The switch to > > > > > > > > > > > > > Jakarta > > > > > > > > > > > > > is > > > > > > > > > > > > > planned for > > > > > > > > > next > > > > > > > > > > > > > year, so I'm still on the 8.x path for the time > > > > > > > > > > > > > being > > > > > > > > > > > > > and > > > > > > > > > > > > > wasn't able > > > > > > > > > to > > > > > > > > > > > > > test if this problem still exists in the 9.x > > > > > > > > > > > > > branch. > > > > > > > > > > > > > > > > > > > > > > > > > > The problem: > > > > > > > > > > > > > On the customers site I'm with the latest TomEE > > > > > > > > > > > > > version > > > > > > > > > > > > > 8.0.15 > > > > > > > > > (running > > > > > > > > > > > > > on Java 11) and everything works fine, because > > > > > > > > > > > > > it is > > > > > > > > > > > > > running in > > > > > > > > > > > > > production mode. But in my development > > > > > > > > > > > > > environment > > > > > > > > > > > > > I'm > > > > > > > > > > > > > still stuck > > > > > > > > > with > > > > > > > > > > > > > 8.0.6, because there I need the development > > > > > > > > > > > > > mode > > > > > > > > > > > > > (javax.faces.PROJECT_STAGE: Development). I > > > > > > > > > > > > > need to > > > > > > > > > > > > > be > > > > > > > > > > > > > able to see > > > > > > > > > > > > > changes in facelets without restarting > > > > > > > > > > > > > everything > > > > > > > > > > > > > everytime, and to > > > > > > > > > get > > > > > > > > > > > > > more detailed error messages. > > > > > > > > > > > > > My attempts to update my dev TomEE to something > > > > > > > > > > > > > newer > > > > > > > > > > > > > than 8.0.6 all > > > > > > > > > > > > > failed, because all (at least all I've tested > > > > > > > > > > > > > so far) > > > > > > > > > > > > > newer versions > > > > > > > > > are > > > > > > > > > > > > > so incredibly slow in dev mode, that it's > > > > > > > > > > > > > unbearable > > > > > > > > > > > > > to > > > > > > > > > > > > > use the > > > > > > > > > > > > > application. Several very looong seconds on > > > > > > > > > > > > > every > > > > > > > > > > > > > request > > > > > > > > > > > > > (even the > > > > > > > > > > > > > little AJAX requests in a view) is simply not a > > > > > > > > > > > > > practical > > > > > > > > > > > > > environment. > > > > > > > > > > > > > After switching to Production mode everything > > > > > > > > > > > > > works > > > > > > > > > > > > > very > > > > > > > > > > > > > fast, and all > > > > > > > > > > > > > my views have response times of very few ms > > > > > > > > > > > > > (even the > > > > > > > > > > > > > big > > > > > > > > > > > > > ones with > > > > > > > > > max. > > > > > > > > > > > > > around 400ms). When switching back to > > > > > > > > > > > > > Development > > > > > > > > > > > > > mode I > > > > > > > > > > > > > have response > > > > > > > > > > > > > times of up to 30s on big views. Not with > > > > > > > > > > > > > 8.0.6, > > > > > > > > > > > > > there > > > > > > > > > > > > > even in > > > > > > > > > > > > > development mode it's reasonable fast in not > > > > > > > > > > > > > getting > > > > > > > > > > > > > higher than 1s. > > > > > > > > > > > > > > > > > > > > > > > > > > My question: > > > > > > > > > > > > > What causes this enormous performance > > > > > > > > > > > > > degradation? > > > > > > > > > > > > > I'm > > > > > > > > > > > > > hoping, it is > > > > > > > > > > > > > just a configuration which now has another > > > > > > > > > > > > > default > > > > > > > > > > > > > value > > > > > > > > > > > > > or the like. > > > > > > > > > If > > > > > > > > > > > > > yes, maybe someone can point me in the right > > > > > > > > > > > > > direction of > > > > > > > > > > > > > where to > > > > > > > > > find > > > > > > > > > > > > > this configuration? If it's not a configuration > > > > > > > > > > > > > thing: > > > > > > > > > > > > > what can I do > > > > > > > > > to > > > > > > > > > > > > > get around this? > > > > > > > > > > > > > > > > > > > > > > > > > > I haven't tested all Versions of TomEE from > > > > > > > > > > > > > 8.0.6 to > > > > > > > > > > > > > 8.0.15, so I > > > > > > > > > can't > > > > > > > > > > > > > say at what version exactly this behaviour > > > > > > > > > > > > > changes. > > > > > > > > > > > > > If it > > > > > > > > > > > > > helps or is > > > > > > > > > > > > > needed, I can make some tests to find the exact > > > > > > > > > > > > > version > > > > > > > > > > > > > where this > > > > > > > > > > > > > happens. Versions I've already tested are: > > > > > > > > > > > > > 8.0.12, > > > > > > > > > > > > > 8.0.13 > > > > > > > > > > > > > and 8.0.15. > > > > > > > > > > > > > All of them are very slow in dev mode. If you > > > > > > > > > > > > > need > > > > > > > > > > > > > more > > > > > > > > > > > > > information, > > > > > > > > > > > > > I'll try to provide it. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance for any help > > > > > > > > > > > > > cu > > > > > > > > > > > > > Jens > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Jonathan | exabr...@gmail.com Pessimists, see a jar as half empty. Optimists, in contrast, see it as half full. Engineers, of course, understand the glass is twice as big as it needs to be.
