[ http://issues.apache.org/jira/browse/XERCESC-1381?page=history ]
Khaled Noaman resolved XERCESC-1381:
------------------------------------
Resolution: Fixed
Hi Christian,
I've committed your fix. Would you please verify? I've looked in the code for
potential side effects, and I think that your fix is valid. When we cache
grammars, we remove it from the grammar resolver table, so I agree with you
that we should not change the table not to adopt elements.
Regards,
Khaled
> Memory Leak in GrammarResolver class
> ------------------------------------
>
> Key: XERCESC-1381
> URL: http://issues.apache.org/jira/browse/XERCESC-1381
> Project: Xerces-C++
> Type: Bug
> Components: Validating Parser (Schema) (Xerces 1.5 or up only)
> Versions: 2.6.0
> Environment: all
> Reporter: Christian Will
> Attachments: GrammarResolver.cpp.patch
>
> Hi,
> the grammar resolver class produces memory leaks under some circumstances. If
> we set the features fgXercesUseCachedGrammarInParse,
> fgXercesCacheGrammarFromParse and lock the grammar pool all new created
> grammars will never be deleted.
> Here is my first example. I create a parser, cache the grammar, lock the
> grammar pool and parse one instance document. I delete the parser object,
> unlock and delete the grammar pool, and terminate the platform utils. After
> all objects are deleted we can see that still 3 kbytes are allocated from the
> grammar pool memory manager.
> I could locate all allocations to [\internal\IGXMLScanner2.cpp(1146)].
> => "fDTDGrammar = new (fGrammarPoolMemoryManager)
> DTDGrammar(fGrammarPoolMemoryManager);"
> Sample Log 1
> ------------
> init...
> parser1 set feature
> SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
> cache grammar... (T:/testfiles/_FIXML/schema/FIXML-main-4-4.xsd)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (5)
> GrammarPool : 4239962 [ 12829334]bytes, 62711 [117134, 70374]blocks (8)
> lock grammarpool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (0)
> GrammarPool : 5501456 [ 12829334]bytes, 97049 [151874, 70776]blocks (0)
> ---------------------------------------------------------------------------
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 50846 [ 308510]bytes, 517 [ 3395, 4370]blocks (40)
> GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (4)
> ---------------------------------------------------------------------------
> delete parser1...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (0)
> unlock grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 4243086 [ 12829334]bytes, 62726 [151889,105114]blocks (0)
> delete grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
> terminate PlatformUtils...
> GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
> done.
> ==-still-allocated-memory-=================================================
> pointer: 3df83c0 size: 44bytes sequence: 151875
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> T:\xerces_cvs\src\xercesc\internal\XMLScanner.cpp(434)
> pointer: 3df86d0 size: 32bytes sequence: 151876
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df87d8 size: 436bytes sequence: 151877
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8a70 size: 512bytes sequence: 151878
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8d58 size: 32bytes sequence: 151879
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df8e60 size: 436bytes sequence: 151880
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df90f8 size: 512bytes sequence: 151881
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df93e0 size: 32bytes sequence: 151882
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df94e8 size: 436bytes sequence: 151883
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9780 size: 512bytes sequence: 151884
> ..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9a68 size: 24bytes sequence: 151885
> T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
> pointer: 3df9b60 size: 12bytes sequence: 151886
> ..\..\..\..\..\src\xercesc/util/XMLString.hpp(1667)
> T:\xerces_cvs\src\xercesc\validators\DTD\XMLDTDDescriptionImpl.cpp(59)
> T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
> T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
> ==-still-allocated-memory-=================================================
> My second example shows the impact a little bit better. I lock the grammar
> pool without loading the grammar. I parse the instance document 3 times. And
> after I delete all objects and terminate the platform utils I can see that
> the associated grammar is still 3 times (11.5MB) in memory.
> Sample Log 2
> ------------
> init...
> parser1 set feature
> SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
> lock grammarpool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
> GrammarPool : 23934 [ 24098]bytes, 597 [ 629, 32]blocks (0)
> ---------------------------------------------------------------------------
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 50862 [ 486594]bytes, 517 [ 3392, 4374]blocks (40)
> GrammarPool : 4265794 [ 12855178]bytes, 63302 [117757, 70406]blocks (5)
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 52466 [ 497174]bytes, 586 [ 6317, 8535]blocks (65)
> GrammarPool : 8120878 [ 16710262]bytes,115564 [224426,140764]blocks (0)
> parse with parser1... (T:/testfiles/_FIXML/order100.xml)
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> Parser One : 54586 [ 498778]bytes, 655 [ 9241, 12695]blocks (144)
> GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
> ---------------------------------------------------------------------------
> delete parser1...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
> unlock grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11953242 [ 20565346]bytes,167249 [331095,211699]blocks (0)
> delete grammarPool...
> PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
> GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
> terminate PlatformUtils...
> GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
> Let me try to explain why this is happening. In GrammarResolver::putGrammar()
> can we see that we put our grammar in the table 'fGrammarBucket' if we don't
> want to cache our grammar or the grammar pool doesn't want cache our grammar
> (because it is locked).
> void GrammarResolver::putGrammar(Grammar* const grammarToAdopt)
> {
> ...
> if (!fCacheGrammar || !fGrammarPool->cacheGrammar(grammarToAdopt))
> {
> // either we aren't caching or the grammar pool doesn't want it
> // so we need to look after it
> fGrammarBucket->put( (void*)
> grammarToAdopt->getGrammarDescription()->getGrammarKey(), grammarToAdopt );
> if (grammarToAdopt->getGrammarType() == Grammar::SchemaGrammarType)
> {
> fGrammarsToAddToXSModel->addElement((SchemaGrammar*)
> grammarToAdopt);
> }
> }
> }
> So far so good. We have all grammars that we don't want to cache or/and can't
> be cached in this table. So normaly if it deletes the table fGrammarBucket
> all grammars will be deleted. But if the feature
> fgXercesCacheGrammarFromParse is set it changes the behavior of the table to
> not adopte the elements. What we can see in
> GrammarResolver::cacheGrammarFromParse().
> void GrammarResolver::cacheGrammarFromParse(const bool aValue)
> {
> reset();
> fCacheGrammar = aValue;
> fGrammarBucket->setAdoptElements(!fCacheGrammar);
> }
> I think that is wrong to change the property of our table to not adopt the
> elements. Whenever we remove an grammar from our list, for example to cache
> to grammar, we call fGrammarBucket->orphanKey(grammarKey), and that means
> already that we remove the grammar wihtout deleting it.
> I removed this line "fGrammarBucket->setAdoptElements(!fCacheGrammar);" made
> some tests and everything looks good.
> But I'm not sure if there are no side effects. And that is the reason why I
> would like to get a second opinion from somebody. I can also send parts of my
> code If you have problems to reproduce the bug.
> Regards,
> Christian Will
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
