[jira] Commented: (XERCESC-1901) memory address leak when running many threads which will parsing xml files
[
https://issues.apache.org/jira/browse/XERCESC-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12793896#action_12793896
]
tao xiaochuan commented on XERCESC-1901:
hello . this ticket can be closed , the reason is found
rpm -qf /lib64/ld-2.5.so
glibc-2.5-24
after using new version
rpm -qf /lib64/ld-2.5.so
glibc-2.5-42
the problem is gone
memory address leak when running many threads which will parsing xml files
--
Key: XERCESC-1901
URL: https://issues.apache.org/jira/browse/XERCESC-1901
Project: Xerces-C++
Issue Type: Bug
Components: SAX/SAX2
Affects Versions: 3.0.1
Environment: Red Hat Enterprise Linux Server release 5.1 (Tikanga)
-g++ --version : x86_64-redhat-linux-g++ (GCC) 4.1.1 20070105 (Red Hat
4.1.1-52)
Reporter: tao xiaochuan
Attachments: mainNew, test.zip
when running 5 threads , in each thread a xml file will (for test purpost
xerces-c-3.0.1\samples\data\long.xml is used)
be parsed, it is found, by top command , VIRT increase large , by pmap
command , 3 (some time 4) memory address block about 64 m
are found , when by 8 threads , there will be 5 (some time 6) such blocks :
Address Kbytes Mode Offset DeviceMapping
2aaab00b9000 64796 - 2aaab00b9000 000:0 [ anon ]
...
2aaab40b9000 64796 - 2aaab40b9000 000:0 [ anon ]
...
2aaab80bb000 64788 - 2aaab80bb000 000:0 [ anon ]
these blocks will not be released even after XMLPlatformUtils::Terminate();
if there is only one thead , there is no such a problem
chief code structure:
main(){
XMLPlatformUtils::Initialize();
//start many threads
//wait for alll threads ends
XMLPlatformUtils::Terminate();
}
in each thread :
while(true){
...
auto_ptrSAXParser parser( new SAXParser() );
parser-setDocumentHandler(this); //the handler do nothing but just
extend from HandlerBase
parser-setErrorHandler(this);//the handler do nothing but just
extend from HandlerBase
parser-setValidationScheme(SAXParser::Val_Never);
parser-setDoSchema(false);
parser-setDoNamespaces(true);
parser-parse(inputFile.c_str());
}
refer to attachment for detail about the simple test code
one more related question :
in the doc , it is said client application needs to Terminate() (or multiple
Terminate() in the case where multiple Initialize() have been invoked before
, does it also suit for more than one threads ?
for example :
thread 1 : XMLPlatformUtils::Initialize(); //A
thread 1 : do some parsing //B
thread 2 : XMLPlatformUtils::Initialize(); //C
thread 2 : do some parsing //D
thread 1 : do some parsing //E
thread 1 : XMLPlatformUtils::Terminate();//F
thread 2 : do some parsing //G
thread 1 : XMLPlatformUtils::Terminate();//H
}
since these two api are all static , then does C will affect E and does F
affect G
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: c-dev-unsubscr...@xerces.apache.org
For additional commands, e-mail: c-dev-h...@xerces.apache.org
[jira] Commented: (XERCESC-1901) memory address leak when running many threads which will parsing xml files
[
https://issues.apache.org/jira/browse/XERCESC-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12791149#action_12791149
]
tao xiaochuan commented on XERCESC-1901:
it is found that the problem happen in only 64 bit system , i have tested two
32 bit system and one 64 bit system (what i can get in company at present)
32 bit system(no such leak problem)
Linux version 2.6.9-42.0.10.ELsmp (r...@yort.fnal.gov) (gcc version 3.4.5
20051201 (Red Hat 3.4.5-2)) #1 SMP Tue Feb 27 08:38:56 CST 2007
g++ (GCC) 3.4.6 20060404 (Red Hat 3.4.6-3)
Copyright (C) 2006 Free Software Foundation, Inc.
32 bit system(no such leak problem)
cat /proc/version
Linux version 2.6.18-128.1.1.el5 (mockbu...@hs20-bc1-5.build.redhat.com) (gcc
version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Mon Jan 26 13:59:00 EST 2009
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.2 (Tikanga)
64 bit system (has such leak problem)
cat /proc/version
Linux version 2.6.18-53.1.13.el5 (brewbuil...@hs20-bc1-6.build.redhat.com) (gcc
version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Mon Feb 11 13:27:27 EST 2008
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.1 (Tikanga)
the test code has also been simpled in one file mainNew.cc (ACE is not used
but pthread),see attachment
i do not know whether this is a problem caused by 64 bit system or a problem
special to xerces-c of 64 bit version
memory address leak when running many threads which will parsing xml files
--
Key: XERCESC-1901
URL: https://issues.apache.org/jira/browse/XERCESC-1901
Project: Xerces-C++
Issue Type: Bug
Components: SAX/SAX2
Affects Versions: 3.0.1
Environment: Red Hat Enterprise Linux Server release 5.1 (Tikanga)
-g++ --version : x86_64-redhat-linux-g++ (GCC) 4.1.1 20070105 (Red Hat
4.1.1-52)
Reporter: tao xiaochuan
Attachments: test.zip
when running 5 threads , in each thread a xml file will (for test purpost
xerces-c-3.0.1\samples\data\long.xml is used)
be parsed, it is found, by top command , VIRT increase large , by pmap
command , 3 (some time 4) memory address block about 64 m
are found , when by 8 threads , there will be 5 (some time 6) such blocks :
Address Kbytes Mode Offset DeviceMapping
2aaab00b9000 64796 - 2aaab00b9000 000:0 [ anon ]
...
2aaab40b9000 64796 - 2aaab40b9000 000:0 [ anon ]
...
2aaab80bb000 64788 - 2aaab80bb000 000:0 [ anon ]
these blocks will not be released even after XMLPlatformUtils::Terminate();
if there is only one thead , there is no such a problem
chief code structure:
main(){
XMLPlatformUtils::Initialize();
//start many threads
//wait for alll threads ends
XMLPlatformUtils::Terminate();
}
in each thread :
while(true){
...
auto_ptrSAXParser parser( new SAXParser() );
parser-setDocumentHandler(this); //the handler do nothing but just
extend from HandlerBase
parser-setErrorHandler(this);//the handler do nothing but just
extend from HandlerBase
parser-setValidationScheme(SAXParser::Val_Never);
parser-setDoSchema(false);
parser-setDoNamespaces(true);
parser-parse(inputFile.c_str());
}
refer to attachment for detail about the simple test code
one more related question :
in the doc , it is said client application needs to Terminate() (or multiple
Terminate() in the case where multiple Initialize() have been invoked before
, does it also suit for more than one threads ?
for example :
thread 1 : XMLPlatformUtils::Initialize(); //A
thread 1 : do some parsing //B
thread 2 : XMLPlatformUtils::Initialize(); //C
thread 2 : do some parsing //D
thread 1 : do some parsing //E
thread 1 : XMLPlatformUtils::Terminate();//F
thread 2 : do some parsing //G
thread 1 : XMLPlatformUtils::Terminate();//H
}
since these two api are all static , then does C will affect E and does F
affect G
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: c-dev-unsubscr...@xerces.apache.org
For additional commands, e-mail: c-dev-h...@xerces.apache.org
[jira] Commented: (XERCESC-1901) memory address leak when running many threads which will parsing xml files
[ https://issues.apache.org/jira/browse/XERCESC-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790307#action_12790307 ] Boris Kolpackov commented on XERCESC-1901: -- I ran your test under valgrind. While there are a number of leaks in it (e.g., you don't free ACE task), none of them are Xerces-C++ related (output below). Can you run valgrind and see what you get? All thread exit ==19096== ==19096== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 9 from 1) ==19096== malloc/free: in use at exit: 19,381 bytes in 24 blocks. ==19096== malloc/free: 30,730 allocs, 30,706 frees, 50,514,225 bytes allocated. ==19096== For counts of detected errors, rerun with: -v ==19096== searching for pointers to 24 not-freed blocks. ==19096== checked 677,216 bytes. ==19096== ==19096== 8 bytes in 1 blocks are still reachable in loss record 1 of 15 ==19096==at 0x4C232DC: operator new(unsigned long) (vg_replace_malloc.c:230) ==19096==by 0x54DB6B6: ACE_TSSACE_Service_Config::TSS_Resources::make_TSS_TYPE() const (TSS_T.cpp:60) ==19096==by 0x54DB4D6: ACE_TSSACE_Service_Config::TSS_Resources::ts_get() const (TSS_T.cpp:219) ==19096==by 0x54DB1DA: ACE_TSSACE_Service_Config::TSS_Resources::operator ACE_Service_Config::TSS_Resources*() const (TSS_T.cpp:53) ==19096==by 0x54DA333: ACE_Service_Config::current_i(ACE_Service_Gestalt*) (Service_Config.cpp:378) ==19096==by 0x54DA196: ACE_Service_Config::current() (Service_Config.cpp:354) ==19096==by 0x54DA35C: ACE_Service_Config::static_svcs() (Service_Config.cpp:394) ==19096==by 0x5542060: ACE_Object_Manager_Preallocations::ACE_Object_Manager_Preallocations() (Object_Manager.cpp:134) ==19096==by 0x55428B2: ACE_Object_Manager::init() (Object_Manager.cpp:257) ==19096==by 0x5542C33: ACE_Object_Manager::ACE_Object_Manager() (Object_Manager.cpp:313) ==19096==by 0x5542E90: ACE_Object_Manager::instance() (Object_Manager.cpp:333) ==19096==by 0x5543D97: ACE_Object_Manager_Manager::ACE_Object_Manager_Manager() (Object_Manager.cpp:758) ==19096== ==19096== ==19096== 64 bytes in 1 blocks are still reachable in loss record 2 of 15 ==19096==at 0x4C232DC: operator new(unsigned long) (vg_replace_malloc.c:230) ==19096==by 0x54DA286: ACE_Service_Config::current_i(ACE_Service_Gestalt*) (Service_Config.cpp:375) ==19096==by 0x54DA196: ACE_Service_Config::current() (Service_Config.cpp:354) ==19096==by 0x54DA35C: ACE_Service_Config::static_svcs() (Service_Config.cpp:394) ==19096==by 0x5542060: ACE_Object_Manager_Preallocations::ACE_Object_Manager_Preallocations() (Object_Manager.cpp:134) ==19096==by 0x55428B2: ACE_Object_Manager::init() (Object_Manager.cpp:257) ==19096==by 0x5542C33: ACE_Object_Manager::ACE_Object_Manager() (Object_Manager.cpp:313) ==19096==by 0x5542E90: ACE_Object_Manager::instance() (Object_Manager.cpp:333) ==19096==by 0x5543D97: ACE_Object_Manager_Manager::ACE_Object_Manager_Manager() (Object_Manager.cpp:758) ==19096==by 0x5543FC4: __static_initialization_and_destruction_0(int, int) (Object_Manager.cpp:774) ==19096==by 0x554400C: global constructors keyed to _ZN18ACE_Object_Manager9instance_E (Object_Manager.cpp:855) ==19096==by 0x5596A45: (within /home/boris/work/ace/ACE-5.5.2/ace/libACE.so.5.5.2) ==19096== ==19096== ==19096== 72 bytes in 1 blocks are still reachable in loss record 3 of 15 ==19096==at 0x4C232DC: operator new(unsigned long) (vg_replace_malloc.c:230) ==19096==by 0x54E2803: ACE_Service_Repository::instance(unsigned long) (Service_Repository.cpp:63) ==19096==by 0x54DCC99: ACE_Service_Gestalt::ACE_Service_Gestalt(unsigned long, bool, bool) (Service_Gestalt.cpp:243) ==19096==by 0x54DA4AE: ACE_Service_Config::ACE_Service_Config(int, unsigned long, int) (Service_Config.cpp:447) ==19096==by 0x54DB40F: ACE_SingletonACE_Service_Config, ACE_Thread_Mutex::ACE_Singleton() (Singleton.inl:14) ==19096==by 0x54DAF9D: ACE_SingletonACE_Service_Config, ACE_Thread_Mutex::instance() (Singleton.cpp:77) ==19096==by 0x54DA084: ACE_Service_Config::global() (Service_Config.cpp:284) ==19096==by 0x54DA18E: ACE_Service_Config::current() (Service_Config.cpp:354) ==19096==by 0x54DA35C: ACE_Service_Config::static_svcs() (Service_Config.cpp:394) ==19096==by 0x5542060: ACE_Object_Manager_Preallocations::ACE_Object_Manager_Preallocations() (Object_Manager.cpp:134) ==19096==by 0x55428B2: ACE_Object_Manager::init() (Object_Manager.cpp:257) ==19096==by 0x5542C33: ACE_Object_Manager::ACE_Object_Manager() (Object_Manager.cpp:313) ==19096== ==19096== ==19096== 80 bytes in 1 blocks are still reachable in loss record 4 of 15 ==19096==at 0x4C232DC: operator new(unsigned long) (vg_replace_malloc.c:230) ==19096==by 0x54DAF90: ACE_SingletonACE_Service_Config, ACE_Thread_Mutex::instance() (Singleton.cpp:77) ==19096==by 0x54DA084:
[jira] Commented: (XERCESC-1901) memory address leak when running many threads which will parsing xml files
[
https://issues.apache.org/jira/browse/XERCESC-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12790526#action_12790526
]
tao xiaochuan commented on XERCESC-1901:
hello . the leak problem we focus on can not be detected by valgrind , since
it is not caused by not using new and delete properly , the memory leak
related to ACE TASK found by valgrind is not the core of the problem , sorry
for that the example
code has made some confusing by it is not a good code
a leak case which can not be found by vargrind :
for example , for thread which is not PTHREAD_CREATE_DETACHED, it will casue
below leak when its parent thread do not join it after it finish :
Address Kbytes Mode Offset
DeviceMapping
40001000 10240 rwx-- 40001000 000:0 [ anon ]
such memory address block is gotten by mmap api , not by new , valgrind can
not detect such leak
for my problem, i have used valgrind already at beginning and also use strace
command to trace mmap calling , but nothing found , then there is the problem
reported
can you repeat the test (for example 5 thread nor more ) and watch it by pmap
( copy xerces-c-3.0.1\samples\data\long.xml to Test1.xml ~ Test10.xml , or
change all TestN.xml to long.xml in main.cc)
and you can also fiind that when there is only one thread , there will be no
such 64 m block no matter how long it will run.
memory address leak when running many threads which will parsing xml files
--
Key: XERCESC-1901
URL: https://issues.apache.org/jira/browse/XERCESC-1901
Project: Xerces-C++
Issue Type: Bug
Components: SAX/SAX2
Affects Versions: 3.0.1
Environment: Red Hat Enterprise Linux Server release 5.1 (Tikanga)
-g++ --version : x86_64-redhat-linux-g++ (GCC) 4.1.1 20070105 (Red Hat
4.1.1-52)
Reporter: tao xiaochuan
Attachments: test.zip
when running 5 threads , in each thread a xml file will (for test purpost
xerces-c-3.0.1\samples\data\long.xml is used)
be parsed, it is found, by top command , VIRT increase large , by pmap
command , 3 (some time 4) memory address block about 64 m
are found , when by 8 threads , there will be 5 (some time 6) such blocks :
Address Kbytes Mode Offset DeviceMapping
2aaab00b9000 64796 - 2aaab00b9000 000:0 [ anon ]
...
2aaab40b9000 64796 - 2aaab40b9000 000:0 [ anon ]
...
2aaab80bb000 64788 - 2aaab80bb000 000:0 [ anon ]
these blocks will not be released even after XMLPlatformUtils::Terminate();
if there is only one thead , there is no such a problem
chief code structure:
main(){
XMLPlatformUtils::Initialize();
//start many threads
//wait for alll threads ends
XMLPlatformUtils::Terminate();
}
in each thread :
while(true){
...
auto_ptrSAXParser parser( new SAXParser() );
parser-setDocumentHandler(this); //the handler do nothing but just
extend from HandlerBase
parser-setErrorHandler(this);//the handler do nothing but just
extend from HandlerBase
parser-setValidationScheme(SAXParser::Val_Never);
parser-setDoSchema(false);
parser-setDoNamespaces(true);
parser-parse(inputFile.c_str());
}
refer to attachment for detail about the simple test code
one more related question :
in the doc , it is said client application needs to Terminate() (or multiple
Terminate() in the case where multiple Initialize() have been invoked before
, does it also suit for more than one threads ?
for example :
thread 1 : XMLPlatformUtils::Initialize(); //A
thread 1 : do some parsing //B
thread 2 : XMLPlatformUtils::Initialize(); //C
thread 2 : do some parsing //D
thread 1 : do some parsing //E
thread 1 : XMLPlatformUtils::Terminate();//F
thread 2 : do some parsing //G
thread 1 : XMLPlatformUtils::Terminate();//H
}
since these two api are all static , then does C will affect E and does F
affect G
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: c-dev-unsubscr...@xerces.apache.org
For additional commands, e-mail: c-dev-h...@xerces.apache.org
