Re: Groovy 3 -> 4 Performance Degradation Sample

[MG]

Hi Jochen,

our code certainly creates a lot of (short lived) class instances*, but 
we do not dynamically create new classes, so the overhead of creating 
the initial Indy callsites should only be incurred once at the 
beginning, and then should no longer play a role. So if the same SQL 
creation code is executed in a loop as in the sample script I provided, 
each loop after the first should be as fast as non-Indy Groovy, no ?

Do you think that a Groovy feature such as e.g. traits could be 
responsible for the constant need for callsite creation ? A less-used 
Groovy feature such as traits, that we use only in certain, more complex 
scenarios, being responsible might explain why the performance problems 
a) occur only in more complex test cases, and b) up to date no other 
project seems to suffer the same problem as we do...

(I just checked the effect of removing the Table.reference(...) calls 
from the script code, since these calls use reflection to create the 
correct Table instance, to see if that has any influence,  but this 
leads to the same amount of speedup for Groovy 3 and 4, so the 
performance degradation between Grooyv 3 and 4 stays the same at 3x)

Cheers,
mg

*Every database table is a Groovy class, with each of the table's 
columns represented by a Column class instance; every table references 
in a SQL expressions is another instatiation of the table's class and 
especially its Column objects (Column objects know their table 
reference, so they cannot be shared with the table root instance), and a 
lot of SQL creation code instantiates multiple new table references.


On 09/05/2022 19:53, Jochen Theodorou wrote:
On 09.05.22 17:41, MG wrote:
Hi Jochen,

since I am not feeling well right now, just some quick answers to some
of your questions:

 1. I am not at liberty to release the source code from our project, not
    even partially, so these JARs are currently all I can provide. I
    felt when I reported this problem initially, there were doubts that
    it even exists, so after I could not pinpoint the problem on my
    side, I invested the time to extract these project parts, clean them
    from any historical references to other modules etc and supply them
    in compiled form.

I think I can test if it is really my assumption or not. I made myself a
little program, that will create a class with many methods and one more
method, that is calling all of them, one-by-one. The methods are empty
and nothing takes parameters. I then put this in the compiler and
measure times for creating the class and running the methods.

For 1000 Methods I get on my machine these results for Groovy-3.0.9
non-Indy:
1 Class: 36ms
10 Classes: 114ms
100 Classes: 985ms
1000 Classes: 8227ms

the last case means, that 1+ million methods are executed inside of the
test class

Then I repeat the test with Groovy-4.0.2 Indy:
1  Class: 137ms        (* 3,80)
10 Classes: 670ms      (* 5,88)
100 Classes: 3711ms    (* 3,77)
1000 Classes: 108235ms (* 13,16)

I would not classify these numbers as a proper test, but it does hint at
what I had assumed... creating the initial callsites in indy is too
expensive

the script I used:

def CLASSES = args[0] as Integer
def METHODS = args[1] as Integer


def methods = ""
def calls   = ""
METHODS.times { i ->
  methods += "\ndef foo$i(){}"
  calls   += "\nfoo$i()"
}

def gcl = new GroovyClassLoader();
def testClasses = (1..CLASSES).collect{
    def testClass = """
    class MyPerfTest$it{
      $methods
      def run() {
        $calls
      }
    }
    """
    gcl.parseClass(testClass)
}


def time1 = System.nanoTime()
for (clazz :testClasses) {
  clazz.newInstance().run()
}
def time2 = System.nanoTime()
def timeDiff = (time2-time1)/1000000
println "Time = $timeDiff ms"

While this is surely not "real world" my goal was testing the callsite
creation, and I think the test does this well enough. Why exactly this
scenario? Because I know indy can perform well in micro-benchmarks,
where you crunch a number in tight loops. and if there is no problem
there, then it must be with the many callsites you tend to visit only
once - like here.

 2. As I have said in previous posts, the performance degradation occurs
    with non-Indy vs Indy. In fact this is how I could pinpoint that
    Indy was to blame: After realizing that our test suite ran for 3h
    under Groovy 4, instead of 1h under Groovy 3, and that this was not
    restricted to a few tests becoming really slow, but was a more or
    less general phenomenon (which was puzzling), I ran the test suite
    under Groovy 3 Indy (for the first time), and got basically the same
    reduced performance.

I had a suspicion for a long time already actually. Bad since I sadly
have barely any time for Groovy these days... and checking this can take
very very long.

[...]
 4. Throwing away the first test loop: It is a bit of an acdemic
    discusssion, since Groovy 4 performance is always 2 to 3 times
    slower in any case. You can execute the test script interleaved
    between Grooyv 3 / 4 multiple times to see how measurements develop,
    but I would say it is clear that Groovy 4 is always way slower than
    Groovy 3 for this test, and that is all that matters :-)

in my little test factor 2-3 times is for me factor 3-4 and that was my
best case.

bye Jochen