I've been trying to run some quick jobs (a check.py not an integrate.py) to try to get insight into this problem. I'm finding (at least on my machine) that forrest is growing to huge size, and getting bogged down in swapping.
I can't say this with certainty, but I feel that both Python and Java degrade very poorly when swapping, not just because they do so much dynamic memory management. I suspect any background thread for garbage collection is getting starved. I don't know if this is true, or even if it matters, but we are getting to a point of resoruce shortage, to the point of outage.
I wonder if Python (Gumpy) and Java (Forrest ) are fighting with each other for resources. Both are hogs, but I am not sure if being a memory hog is cause or effect. I don't know if growth is occuring 'cos we finally added the last straw to the camel's back, or if we have some wierd loop bug. I suspect forrest is getting stuck on a certain page, but I have no insight into what is causing this -- nor if the cause is somehow Gumpy. Doing a much smaller Gump run, hence smaller xdocs set, works w/o problem.
When doing a test run here (without even doign the build parts) I see Gumpy (the python instance) growing to 136M! I have no clue as to why it would do that! I wish I could get inside the Python instance to understand where the memory is, or if it is being leaked. Maybe this is just crowding out forrest.
Whatever the cause, I am really starting to get 'done' with forrest. I support it's use, I introduced it & have dealt with the issues and built workarounds from day one, but it is hard work w/ no fun. I feel it (through Gumpy trying to dynamically create xdocs) has been the weak/slow link in Gumpy for a while.
I suspect even the forrest developers here might agree that I took this too far. Forrest is great for sites, but the gump outputs are getting huge and not really benefiting from Forrest skins, etc. Maybe I am wrong, maybe I need to approach the forrest developers (mail to their team) to see what they think on this.
I wonder if we ought consider replacing Forrest with a pure Python HTML producer. As above, I can't prove that forrest is the problem, but a pure Python solution might just halve the unknowns.
Two possible solutions here:
1) use forrest as a dynamic application 2) have HTML generate by python
I would go for 1) since it would keep us the ability to do dynamic stuff like metadata manipulation.
I volunteer to setup 1)
-- Stefano.
smime.p7s
Description: S/MIME Cryptographic Signature
