Hi we have some memory problems concerning images
I have been running some tests and it seams there are 2 problems with the
current 1.6 relase of mmbase (see TESTS in this message)
MagicFile still s*cks. it eats memory (luckely it gives it back after a while
but when the server is bussy this is enough to get OutOfMemory errors
secondly
Node node = cloud.getNode(99)
results in
SELECT * FROM install_images WHERE number=99;
With mysql this results in returning all the data (even the handle fiels)
(causing out of memmory errors / maybe in the resultset/maybe in the node
cache)
and
node.getByteValue("handle")
results in
SELECT handle FROM install_images WHERE number=99
once a byte[] has been loaded it is kept in memmory
While testing I have made some modifications to the MMObjectBuilder
(anywhere where there was a "SELECT *" I changed this to
"SELECT " +getNonByteArrayFields(tableName)
This had great inpact . i can list 1000 images with no problem (even if the
node cache is set to 4000 nodes).
Is this something we want in 1.6 ?
I can't work on the 1.7 branch because it's broken ( at least on mysql and
postgress) but how wil this problem be fixed in 1.7 ?
I would like to know if you think this i a good idea?
(Images.java has it's onw handle cache for small images)
secondly the MagicFile. for the moment I have disabled it for Images
for the code is so "complex" that I realy don't have a clue how to fix it.
I would like to make the MagicFile configurable so that it's possible to use
the unix "file" command.
__START TESTS__
The test consists of importing images using RMMCI (a set of 111 images being
34 MB of data)
tests 1:
no security
node cache (cache 10 nodes)
no MagicFile
import 1733 node at 5.198776758409786 nodes/s
(530 MB data with no problems)
test 2:
no security
node cache (cache 4000 nodes)
no MagicFile
import 205 nodes at 4.761904761904762 nodes/s
Exception in thread "main" org.mmbase.bridge.BridgeException: Error occurred
in server thread; nested exception is:
java.lang.OutOfMemoryError
at
org.mmbase.bridge.remote.implementation.RemoteNode_Impl.commit(RemoteNode_Impl.java:72)
test 3:
security
node cache (cache 10 nodes)
MagicFile
import 100 nodes at 0.37735849056603776 nodes/s
memory full but no out of memory..
test 4:
security
node cache (cache 10 nodes)
MagicFile
2 inpurt threads..
import 10 nodes
Exception in thread "main" org.mmbase.bridge.BridgeException: Error occurred
in server thread; nested exception is:
java.lang.OutOfMemoryError
at
org.mmbase.bridge.remote.implementation.RemoteNode_Impl.commit(Remo
__END TESTS__
--
Kees Jongenburger
Mediapark C101 Hilversum
+31 (0)35 6772910
