Re: [Open Babel] Openbabel in Python in concurrent processes
Did you see this article? You didn't say what Linux you are using, but if it's Ubuntu, this may be of interest. http://blog.xen.org/index.php/2011/11/29/baremetal-vs-xen-vs-kvm-redux/ Craig On Fri, Jul 20, 2012 at 1:01 PM, Maciek Wójcikowski wrote: > I also thought about L1 and L2 caches. But underlying problem seams to be > much simpler. The server is VM on modern AMD server (4x 12 core opteron + > 96gigs of DDR3 ECC ram). Due to virtualisation heavy memory IO (memory > usage was less tan 50% including caches) minimization was running so slow. > I've done same test on Intel Desktop (i7-2600) and process execution time > changed from 18 min (single proc.) to 21 min (8 concurrent proc.). I'd say > that the bias is due to Intel Turbo technology and HT (4 physical cores). > > The bottom line should be: don't use virtualisation (at least Xen) for > memory intensive computing. I must say that I'm pretty disappointed with > those results, although it's quite obvious... > > Out of curiosity I'm going to rerun the tests on KVM and probably on > OpenVZ to see how it goes there. > > Pozdrawiam, | Best regards, > Maciek Wójcikowski > mac...@wojcikowski.pl > > > 2012/7/20 Craig James > >> >> >> On Fri, Jul 20, 2012 at 2:16 AM, Maciek Wójcikowski < >> mac...@wojcikowski.pl> wrote: >> >>> Hi, >>> I'm doing some scripting in Python to minimize ligands and want to run >>> them concurrently on SMP server (many processes, one core per proc.). The >>> problem that I hit is that when I run one instance, process take 100% CPU >>> and runs for ~25min. For the same set if I run 32 concurrent processes it >>> runs from 60-130 min. Why is that? Server has 32 phisical cores (AMD >>> Opteron) so there is no turbo mode, like in intel, to produce such power >>> increase. Script is not doing any IO (at least I don't know of it, it gets >>> data from MySQL, doing only SELECTs). >>> >>> How can I debug this issue? Any thoughts? >>> >> >> There are many reasons why this might happen. You should do an >> experiment starting with two process, then four, eight, etc ... and see >> where it starts to slow down. >> >> You say 32 CPUs, but now many CPU chips? Is this 4x8-core CPUs, >> 8x4core, or something else? >> >> When you run each process, how much memory does it use? How much memory >> do you have? >> >> My first guess is that you are running into problems with memory >> contention -- processors fighting each other for memory. This can be >> either cache memory (for small processes) or main memory (if the ligand >> minimization is a very large process). >> >> Modern CPUs are *much* faster than modern memory, which is why they have >> multi-level (L1 and L2) caches (see >> http://en.wikipedia.org/wiki/CPU_cache), and also have "NUMA" memory >> architecture (see http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access). >> When you run a single process, it gets full access to all cache memory and >> all main memory. When you run up to N processes where N is the number of >> CPU chips (not cores), then each CPU has full access to its caches, but >> there may be contention for main memory. When you run N processes where N >> is the total number of CPUs (in your case, 32), then each CPU has to share >> access to both main memory and to the on-chip memory caches and data >> channels. >> >> The only time a process will scale from 1 to 32 on a 32-CPU system is if >> each process is compute-bound and uses so little memory that all 32 >> processors are mostly working from their internal caches and don't ever >> have to wait for memory access. >> >> If you want to read a long but fascinating article about how this affects >> other projects, check this one out. It's about MySQL running very slowly >> on very fast CPUs because of memory contention. >> >> >> http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/ >> >> >> Craig >> >> >> >>> >>> Pozdrawiam, | Best regards, >>> Maciek Wójcikowski >>> mac...@wojcikowski.pl >>> >>> >>> -- >>> Live Security Virtual Conference >>> Exclusive live event will cover all the ways today's security and >>> threat landscape has changed and how IT managers can respond. Discussions >>> will include endpoint security, mobile security and the latest in malware >>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>> ___ >>> OpenBabel-discuss mailing list >>> OpenBabel-discuss@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/openbabel-discuss >>> >>> >> > -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/
Re: [Open Babel] Openbabel in Python in concurrent processes
I also thought about L1 and L2 caches. But underlying problem seams to be much simpler. The server is VM on modern AMD server (4x 12 core opteron + 96gigs of DDR3 ECC ram). Due to virtualisation heavy memory IO (memory usage was less tan 50% including caches) minimization was running so slow. I've done same test on Intel Desktop (i7-2600) and process execution time changed from 18 min (single proc.) to 21 min (8 concurrent proc.). I'd say that the bias is due to Intel Turbo technology and HT (4 physical cores). The bottom line should be: don't use virtualisation (at least Xen) for memory intensive computing. I must say that I'm pretty disappointed with those results, although it's quite obvious... Out of curiosity I'm going to rerun the tests on KVM and probably on OpenVZ to see how it goes there. Pozdrawiam, | Best regards, Maciek Wójcikowski mac...@wojcikowski.pl 2012/7/20 Craig James > > > On Fri, Jul 20, 2012 at 2:16 AM, Maciek Wójcikowski > wrote: > >> Hi, >> I'm doing some scripting in Python to minimize ligands and want to run >> them concurrently on SMP server (many processes, one core per proc.). The >> problem that I hit is that when I run one instance, process take 100% CPU >> and runs for ~25min. For the same set if I run 32 concurrent processes it >> runs from 60-130 min. Why is that? Server has 32 phisical cores (AMD >> Opteron) so there is no turbo mode, like in intel, to produce such power >> increase. Script is not doing any IO (at least I don't know of it, it gets >> data from MySQL, doing only SELECTs). >> >> How can I debug this issue? Any thoughts? >> > > There are many reasons why this might happen. You should do an experiment > starting with two process, then four, eight, etc ... and see where it > starts to slow down. > > You say 32 CPUs, but now many CPU chips? Is this 4x8-core CPUs, 8x4core, > or something else? > > When you run each process, how much memory does it use? How much memory > do you have? > > My first guess is that you are running into problems with memory > contention -- processors fighting each other for memory. This can be > either cache memory (for small processes) or main memory (if the ligand > minimization is a very large process). > > Modern CPUs are *much* faster than modern memory, which is why they have > multi-level (L1 and L2) caches (see http://en.wikipedia.org/wiki/CPU_cache), > and also have "NUMA" memory architecture (see > http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access). When you run a > single process, it gets full access to all cache memory and all main > memory. When you run up to N processes where N is the number of CPU chips > (not cores), then each CPU has full access to its caches, but there may be > contention for main memory. When you run N processes where N is the total > number of CPUs (in your case, 32), then each CPU has to share access to > both main memory and to the on-chip memory caches and data channels. > > The only time a process will scale from 1 to 32 on a 32-CPU system is if > each process is compute-bound and uses so little memory that all 32 > processors are mostly working from their internal caches and don't ever > have to wait for memory access. > > If you want to read a long but fascinating article about how this affects > other projects, check this one out. It's about MySQL running very slowly > on very fast CPUs because of memory contention. > > > http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/ > > > Craig > > > >> >> Pozdrawiam, | Best regards, >> Maciek Wójcikowski >> mac...@wojcikowski.pl >> >> >> -- >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. Discussions >> will include endpoint security, mobile security and the latest in malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> ___ >> OpenBabel-discuss mailing list >> OpenBabel-discuss@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/openbabel-discuss >> >> > -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ OpenBabel-discuss mailing list OpenBabel-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openbabel-discuss
Re: [Open Babel] Problem with gen3D
> Thanks for the reply - this problem occurs in the builder.Build method > before the force field optimization, where initial coordinates are > given to each atom. It's not so much an issue of the "constraints" as the fact that you have a messy fused-ring system. The ring perception and particularly aromatization routines perform very poorly on systems with large numbers of fused rings. I'm currently coding a distance-geometry 3D coordinate generation algorithm, but it will probably be another few days before I have anything particularly useful. I'm saving your e-mail as an example that should work quickly and I'll send you a message if you'd like to try out the development code. -Geoff -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ OpenBabel-discuss mailing list OpenBabel-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openbabel-discuss
Re: [Open Babel] Problem with gen3D
Thanks for the reply - this problem occurs in the builder.Build method before the force field optimization, where initial coordinates are given to each atom. -Peter On Fri, Jul 20, 2012 at 8:02 AM, scott_m wrote: > Gen3d does a steepest descent optimization, weighted rotor search and then a > conjugate gradients optimization (see > http://forums.openbabel.org/what-does-gen3d-do-under-the-hood-td3397296.html). > You might want to try a simple forcefield optimization to start with (like > steepest descent) and see how that works. > > > > > -- > View this message in context: > http://forums.openbabel.org/Problem-with-gen3D-tp4655296p4655300.html > Sent from the General discussion mailing list archive at Nabble.com. > > -- > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > ___ > OpenBabel-discuss mailing list > OpenBabel-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/openbabel-discuss -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ OpenBabel-discuss mailing list OpenBabel-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openbabel-discuss
Re: [Open Babel] Problem with gen3D
Gen3d does a steepest descent optimization, weighted rotor search and then a conjugate gradients optimization (see http://forums.openbabel.org/what-does-gen3d-do-under-the-hood-td3397296.html). You might want to try a simple forcefield optimization to start with (like steepest descent) and see how that works. -- View this message in context: http://forums.openbabel.org/Problem-with-gen3D-tp4655296p4655300.html Sent from the General discussion mailing list archive at Nabble.com. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ OpenBabel-discuss mailing list OpenBabel-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openbabel-discuss
Re: [Open Babel] Openbabel in Python in concurrent processes
On Fri, Jul 20, 2012 at 2:16 AM, Maciek Wójcikowski wrote: > Hi, > I'm doing some scripting in Python to minimize ligands and want to run > them concurrently on SMP server (many processes, one core per proc.). The > problem that I hit is that when I run one instance, process take 100% CPU > and runs for ~25min. For the same set if I run 32 concurrent processes it > runs from 60-130 min. Why is that? Server has 32 phisical cores (AMD > Opteron) so there is no turbo mode, like in intel, to produce such power > increase. Script is not doing any IO (at least I don't know of it, it gets > data from MySQL, doing only SELECTs). > > How can I debug this issue? Any thoughts? > There are many reasons why this might happen. You should do an experiment starting with two process, then four, eight, etc ... and see where it starts to slow down. You say 32 CPUs, but now many CPU chips? Is this 4x8-core CPUs, 8x4core, or something else? When you run each process, how much memory does it use? How much memory do you have? My first guess is that you are running into problems with memory contention -- processors fighting each other for memory. This can be either cache memory (for small processes) or main memory (if the ligand minimization is a very large process). Modern CPUs are *much* faster than modern memory, which is why they have multi-level (L1 and L2) caches (see http://en.wikipedia.org/wiki/CPU_cache), and also have "NUMA" memory architecture (see http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access). When you run a single process, it gets full access to all cache memory and all main memory. When you run up to N processes where N is the number of CPU chips (not cores), then each CPU has full access to its caches, but there may be contention for main memory. When you run N processes where N is the total number of CPUs (in your case, 32), then each CPU has to share access to both main memory and to the on-chip memory caches and data channels. The only time a process will scale from 1 to 32 on a 32-CPU system is if each process is compute-bound and uses so little memory that all 32 processors are mostly working from their internal caches and don't ever have to wait for memory access. If you want to read a long but fascinating article about how this affects other projects, check this one out. It's about MySQL running very slowly on very fast CPUs because of memory contention. http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/ Craig > > Pozdrawiam, | Best regards, > Maciek Wójcikowski > mac...@wojcikowski.pl > > > -- > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > ___ > OpenBabel-discuss mailing list > OpenBabel-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/openbabel-discuss > > -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ OpenBabel-discuss mailing list OpenBabel-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openbabel-discuss
Re: [Open Babel] about usage of openBabel
Ozgun, On Thu, Jul 19, 2012 at 8:18 PM, ozgun bursali wrote: > I am a computer engineering student from Turkey and I am doing my > internship right now in Arizona State University. As a part of my work I > need to convert mol files to inchi files. I searched in google and read > many articles about this topic and I found your programme. Thanks for > sharing that. Anyway my question is how to I use it in my project and also > Can I use it because this is a research project for the class. > Welcome to the OpenBabel community. OpenBabel is a free program released under the GNU Public License (GPL) version 2. That means you can use it for anything you like, for free. See this web site: http://www.gnu.org/licenses/gpl-2.0.html If you have questions, send email to this forum (not to the individual members) and you will almost always get a reply the same day. Good luck with your project. Craig -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ OpenBabel-discuss mailing list OpenBabel-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openbabel-discuss
[Open Babel] about usage of openBabel
Dear Sir or Madam I am a computer engineering student from Turkey and I am doing my internship right now in Arizona State University. As a part of my work I need to convert mol files to inchi files. I searched in google and read many articles about this topic and I found your programme. Thanks for sharing that. Anyway my question is how to I use it in my project and also Can I use it because this is a research project for the class. Thanks for your time and consideration Ozgun Bursali -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ OpenBabel-discuss mailing list OpenBabel-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openbabel-discuss
[Open Babel] Problem with gen3D
I've been having trouble with the 3D coordinate generation in Open Babel. The generator hangs on particular molecules with complicated bonding, and my guess is that the code gets stuck in an infinite loop when it can't satisfy all the constraints while building a 3D structure. For example the following molecule fragment, consisting of two merged Boron icosahedrons, appears to run forever. obabel -:"[Pd]1234567[B]89%10%11%12%13[B]%14%15%16%17%181[B]1%192[B]2%203[B]348[B]48%20[B]%20%192[B]2%151[B]19%14([B]%1034[B]8%2021)[B]12%12%17[B]34%13[B]85%11[B]596[B]67%16[B]7%109[B]385[B]24%10[B]%18167" -o sdf -O temp.sdf --gen3D I've tried working around this problem by running gen3D on each molecule as it's own subprocess, but this has become annoying. My next step is to modify the source code to avoid this problem, or at least add a timeout option so that the code returns. I was wondering if anyone else on this list was having the same problems, and if anyone had any ideas. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ OpenBabel-discuss mailing list OpenBabel-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openbabel-discuss
[Open Babel] Openbabel in Python in concurrent processes
Hi, I'm doing some scripting in Python to minimize ligands and want to run them concurrently on SMP server (many processes, one core per proc.). The problem that I hit is that when I run one instance, process take 100% CPU and runs for ~25min. For the same set if I run 32 concurrent processes it runs from 60-130 min. Why is that? Server has 32 phisical cores (AMD Opteron) so there is no turbo mode, like in intel, to produce such power increase. Script is not doing any IO (at least I don't know of it, it gets data from MySQL, doing only SELECTs). How can I debug this issue? Any thoughts? Pozdrawiam, | Best regards, Maciek Wójcikowski mac...@wojcikowski.pl -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ OpenBabel-discuss mailing list OpenBabel-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/openbabel-discuss
