TemplateEngines and Out of Memory
Hi, we are using Groovy in our web-application. There we create a "lot" of
dynamic String and are handling that with the TemplateEngine as we need more
complex expressions there. Now we noticed that we are running into a kind of
weird OOM situation. I could narrow the problem down to a quite simple
reproducible example:
package kos.tools.template;import java.util.HashMap;import
java.util.Map;import javax.script.Bindings;import
javax.script.ScriptEngine;import javax.script.ScriptEngineManager;import
javax.script.SimpleBindings;public class Test { public static void
main(String[] args) throws Exception { ScriptEngineManager factory =
new
ScriptEngineManager(); ScriptEngine engine =
factory.getEngineByName("groovy"); String template
= "step-${i}";
String groovy = "def engine = new
groovy.text.GStringTemplateEngine();\n"
+ "def res =
engine.createTemplate(template).make(bindings);\n" +
"return res.toString();"; for (int i = 0; i < (1000); i++) {
Bindings
vars = new SimpleBindings();vars.put("template", template);
Map templateObjects = new HashMap<>();
vars.put("bindings", templateObjects);
templateObjects.put("i", i);
Object res = engine.eval(groovy, vars);
if (i % 100 == 0) {
System.out.println("->" + res); } } }
}
Running that example with an Oracle JDK 1.8 and a quite small memory (just
for keeping the running time small) of 32M leads to an OOM at around 2500
steps. I opened up a disussion on Stackoverflow where John Vint gave me some
hints ( see here
) and it might be also a bug of the GarbageCollector itself. But in essence
I think there should be a way to use the TemplateEngine in a stable way. I
am happy for any suggestions and examples of how to solve this situation.
Tino
--
View this message in context:
http://groovy.329449.n5.nabble.com/TemplateEngines-and-Out-of-Memory-tp5733514.html
Sent from the Groovy Users mailing list archive at Nabble.com.
GenericsVisitor change for Groovy 2.5.0
Hi everyone, I moved the GenericsVisitor earlier in the compilation process. This means that errors involving class usage but supplying an incorrect number of type parameters, e.g. List or HashMap or invalid Diamond operator usage will now be flagged earlier. This is nice because we had other parts of the compiler which used to run prior to the generics visitor that assumed that such information was correct and would fail with an ArrayIndexOutOfBoundsException. In those circumstances we now get a nicer error message. The flip side is that if anyone had a DSL that relied on such invalid syntax and had a transform that corrected the errors that ran in the early stages of the compiler, then it will likely now run after the GenericsVisitor which means the error will already be flagged. I am not aware of any such usage of invalid Groovy syntax being relied upon but if anyone is in that position, now is the time to let us know. You can still use a global transform but there might be other workarounds possible if you let us know the circumstances. For further details see: https://issues.apache.org/jira/browse/GROOVY-7865 Cheers, Paul.
Re: Is it possible to enable CompileStatic for an entire project
On 24.06.2016 18:12, Mr Andersson wrote: [...] static Groovy is on par most of the time, dynamic Groovy depends, but I think it is more in the range of being 2.5 times java (100ms runtime becomes 250ms). I think that is close enough and there is still some potential But a Java fork would be almost 1:1, if all it generated was Java like code. Java like code means to do what Java does. Basically you want a language that includes Java and does things different only in parts, which are not in Java... That means loosing ==, that means loosing our array access logic, that means loosing double dispatch of course.. doing builders will become difficult... unless you want to do them like Kotlin. Runtime meta programming basically impossible I think the performance issues is the invokeDynamic calls that groovy does internally, and they might not necessarily be needed. there not that many of those, really. [...] However, any editor could in principle, and in theory tell you on any call, what exceptions can be thrown and what kind of exceptions you could choose to handle. Ofcourse that would result in many exceptions often but the need to declare them on each method is largely syntatical and ugly. think of a public method and an unknown subclass overriding that method. How can your editor know, that your library method can only throw the exceptions you use in your code and no other? It cannot, unless you make the method final, or say you don´t do libraries ;) bye Jochen
Re: Integrating Groovy with a Java EE application and Maven
On 24.06.2016 17:41, Mr Andersson wrote: [...] If you want to convince me of switching, i am still looking/waiting for someone to offer cyclic dependency support between modules, since without it, my code is getting more complex than need be. then you probably do not want modules, but just multiple source folders... because what advantages are left of modules if they are cyclic? [...] I also have hate the fact that I have to keep creating modules for every thing. If I have a module Core containing many utility classes, another module should be able to import only a package from Core, alternatively only a class from Core. That might be tricky, but it is desirable, and not unresolvable. Take the case where I decide to publish an Opensource library using a few things out of Core. Now I have to create another module move things out of Core, to expose and share only what is required with world. If another opensource library is created that uses something else from Core, and something from the already extracted module, then we end up in hell. you can filter sources/classes by (package)name... even create virtual projects in gradle bye Jochen
Re: @compileStatic and groovy.sql
I've been playing around with sql2o actually.
http://www.sql2o.org/
it has very good performance for read. https://github.com/aaberg/sql2o
and 99% of my SQL needs are read only.
On Fri, Jun 24, 2016 at 9:34 AM, Cédric Champeau
wrote:
> For type safe sql those days I would recommend to use jOOQ.
> Le 23 juin 2016 06:58, "Wilson MacGyver" a écrit :
>
>> Ouch I will continue to embrace dynamic then :)
>>
>> On Wed, Jun 22, 2016 at 1:56 PM Guillaume Laforge
>> wrote:
>>
>>> Perhaps a custom type checker extension could be fed with the jdbc
>>> metadata...
>>> Le 22 juin 2016 7:41 PM, "Shil Sinha" a écrit :
>>>
Hi Marc,
You could use the map access syntax i.e. foo['id'] instead and
cast/coerce the result to the appropriate type.
On Wed, Jun 22, 2016 at 1:37 PM Wilson MacGyver
wrote:
> Hi,
>
> If I want to use compileStatic with groovy.sql, how would I do that?
>
>
> the problem as far as I can tell is
>
> sql.eachRow("select id, from whatever") { foo ->
> ...
> foo.id
> }
>
> returns data that is known at runtime
>
> but at compile time. there is no way to know that the SQL statement
> returns a
> id column.
>
> is there a way to do it?
>
> Thanks,
> Mac
>
> --
> Omnem crede diem tibi diluxisse supremum.
>
--
Omnem crede diem tibi diluxisse supremum.
Re: Is it possible to enable CompileStatic for an entire project
On 06/22/2016 07:45 PM, Jochen Theodorou wrote:
On 22.06.2016 18:55, Mr Andersson wrote:
[...]
We, at our company can only ship groovy if it is fully static, no way to
manipulate. It's difficult to hammer it through otherwise. Big
enterprise JEE companies don't take risks, especially when the tools are
handed over to 5 000 dev across the globe.
We have to know that nobody will start utility the crazy ( but neat )
features of the dynamic parts of Groovy, but we like that it is such a
powerful language, and easy to learn for any java developer that it
would be a smash hit.
yeah, I have a lot of plans in a direction that allows more control
about the runtime meta programming and actually allows classes to
immunize themselves against meta class changes. The basic concept and
the ideas are there, the detail work and a good plan to migrate
current Groovy to that version is very difficult to do.
[...]
I had alot of useful code that I can never use or pull out, because it's
just to dynamic. Over time one starts to realize how important it is to
know at compile time that things will work.
do you have a different Java than I have? No seriously, even with the
infamous Haskell I would not assume that my program is correct just
because it compiles.
[...]
I have been in love with this language, and even wrote my masters thesis
about it.
really?about what exactly?
Well, it was a 140 pages about Groovy language compared to the Java
language as well as Grails. Basically I went through all of the new
things language Groovy provided that was lacking in Java and how it was
more consice, expressive and dangerously powerful in the wrong hands.
Grails I went through basically the same things. It was around 70 pages
on each. Pretty quick guide to both actually. Nothing research worthy
maybe, but hey it was only a thesis :P
It has so much potential, even to replace Java. It's power is it's
syntax and it's closeness to Java syntax.
Adoption in the Java community would be massive if it were more static.
And you tell that a person that did hear about people not accepting
the language already just because of the name. I know about cases in
which a company did not use a library because it is written in
Groovy... just because it is written in Groovy. All that would not
change with static compilation by default at all.
Yes, personally I like the name, but a couple of times i've mentioned it
to people that haven't heard about it and I realize how "stupid" the
name sounds and how unserious what I am proposing sounds like.
Maybe it's time for a completely new name, possibly even a new
foundation. Just start over with JDK 9, and start forking and
implementing Groovy syntax first on top of it, like elvis, safe nav,
closure syntax, multiline strings, GStrings, default public, no
semicolon, no parantheses, scripting capability, default getters and
settings, default constructors. That to me is Java with some elegant
syntax.
Then you could move to providing a safe and static AST injections of
things. public SomeClass Overrides java.util.ArrayList {...}, seems like
neat syntax to me. Then one could read them at time compile time, and
replace the original ArrayList on a per jar file. If a jar imports or
uses another modified module it would have access to those added methods
for instance. It can still override methods just for it's own use.
One shouldn't be able to override another libraries function. I think
that's just dangerous and rarely makes any sense.
[...]
I am not sure how well, performance wise Groovy does compared with Java,
Scala, Kotlin, Ceylon and so on. If it is close, then I say just offer
us a static version with what you have. You are so close now.
static Groovy is on par most of the time, dynamic Groovy depends, but
I think it is more in the range of being 2.5 times java (100ms runtime
becomes 250ms). I think that is close enough and there is still
some potential
But a Java fork would be almost 1:1, if all it generated was Java like
code. I think the performance issues is the invokeDynamic calls that
groovy does internally, and they might not necessarily be needed.
But I would still like to argue that maybe Groovy needs to be rewritten
from scratch in order to perform even better. But I know little about
this and have little experience with the @CompileStatic annotations
because I have been away from this language for so long.
For better performance of dynamic Groovy we need to rework the MOP.
For example not using exceptions for missing properties and such. But
I say this for too long already.
Yes, this also got me thinking of another thing. The difference between
runtime and checked exceptions in Java or other languages actually does
not make any sense at all.
In Java if a method throws a checked exception, it can either handle it
right there, or throw it. Most exceptions is to be bubbled up, to the
original caller. If a user cal
Re: Integrating Groovy with a Java EE application and Maven
On 06/22/2016 10:30 AM, Cédric Champeau wrote:
You should really give Gradle a try. It's pretty straightforward.
I have tried it, i found it "too dirty", too groovy, too dynamic,
although I know it's an excellent tool for building and hooking into the
build process.
I also found there was too many ways of doing the same thing, which to
me always is an issue, since I have to figure out what the difference
is, and then choose which way is most preferable. Too much choice leaves
my mind spinning.
But, I am not a great fan of maven either.
If you want to convince me of switching, i am still looking/waiting for
someone to offer cyclic dependency support between modules, since
without it, my code is getting more complex than need be.
For me the problem can be solved, since packages in a Java project can
have cyclic dependencies.
One just need a way to declare rules that prevent one package to access
another if it is not declared that it wants access to it. Basically,
"modules/packages" would have a rule saying that it can access another
package.
A build tool such as maven, gradle, "new one" could then build one jar
containing the other resources, mimicking one big module containing
other "modules". I know Intelij modules allows for cyclic dependencies
too, so the problem is defenitely solvable, although Intellij has a
habit of destroying whatever setup one has, and there is no way to
externalize that outside of Intellij.
All modules could then access each other as per definition as they
please and we would have cyclic dependency.
I know people argue that modules shouldn't have cyclic dependencies and
that is bad design. What is true however, is the opposite. Bad design is
forcing one not to have it, just out of a principle. There are cases
where cyclic dependency is important, and a problem becomes almost
impossible if not awkward to solve without.
I also have hate the fact that I have to keep creating modules for every
thing. If I have a module Core containing many utility classes, another
module should be able to import only a package from Core, alternatively
only a class from Core. That might be tricky, but it is desirable, and
not unresolvable. Take the case where I decide to publish an Opensource
library using a few things out of Core. Now I have to create another
module move things out of Core, to expose and share only what is
required with world. If another opensource library is created that uses
something else from Core, and something from the already extracted
module, then we end up in hell.
Maybe Gradle is able to pull that off with build scripts though. Hmm. It
got me thinking of a switch might give me the power to do what I want.
I am glad I did everything in Maven so far, because it gave me control,
and I guess I could convert to Gradle now.
Example of cyclic dependency issues, forcing:
module App
module FinanceApp -> access to -> App
module TvApp -> access to -> App
module NewsApp -> access to -> App
But what if you wanted a central location where you did something that
needs access to all modules?
For example, I prefer to have all exception handling in one location, as
well as all Authentication and Permissions declared in one class, with
access FinanceController.class for instance ( class reference, not
strings to be refactor friendly) as to not have them declared across
many modules and loosing control of who has access to what. I also need
to build bundles for all the modules, preferably in one location.
The perfect location would be App. But App can't access Finance, Tv or
News now.
So to resolve it, maven and other build tools forces you to declare
another module, which we can call
module Base -> access to >{ Finance, Tv, News } -> App
The others would then:
module FinanceApp -> access to -> App
module TvApp -> access to -> App
module NewsApp -> access to -> App
One extra module just to be able to support access from one module to
all. But now the problem arises where Finance can't see what is being
configured in Base. For instance, in development mode, the finance
application might need to rebuild the bundles or reconfigure the
authentication logic. The problem is solvable through setting a lambda
for that in App from Base, but it's not a pretty solution.
Cyclic dependency makes sense in packages, why not across modules, if I
desire it.
Le 21 juin 2016 23:40, "Mr Andersson" > a écrit :
On 06/21/2016 10:39 PM, Jochen Theodorou wrote:
On 21.06.2016 09:04, Mr Andersson wrote:
Gmaven or Gmaven 2 did not work for me either. Resulted in a
bunch of
compilation issues which I started to correct, but then gave up
on. I
shouldn't have to change my code to get on Groovy.
yeah, forget about those... gmavenplus is supposed to work. If
that one does not do the job, then there is a problem
[...]
Plus have you seen the size of
Re: @compileStatic and groovy.sql
For type safe sql those days I would recommend to use jOOQ.
Le 23 juin 2016 06:58, "Wilson MacGyver" a écrit :
> Ouch I will continue to embrace dynamic then :)
>
> On Wed, Jun 22, 2016 at 1:56 PM Guillaume Laforge
> wrote:
>
>> Perhaps a custom type checker extension could be fed with the jdbc
>> metadata...
>> Le 22 juin 2016 7:41 PM, "Shil Sinha" a écrit :
>>
>>> Hi Marc,
>>>
>>> You could use the map access syntax i.e. foo['id'] instead and
>>> cast/coerce the result to the appropriate type.
>>>
>>> On Wed, Jun 22, 2016 at 1:37 PM Wilson MacGyver
>>> wrote:
>>>
Hi,
If I want to use compileStatic with groovy.sql, how would I do that?
the problem as far as I can tell is
sql.eachRow("select id, from whatever") { foo ->
...
foo.id
}
returns data that is known at runtime
but at compile time. there is no way to know that the SQL statement
returns a
id column.
is there a way to do it?
Thanks,
Mac
--
Omnem crede diem tibi diluxisse supremum.
>>>
