Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
Excellent points Ken; thanks! I expanded the FAQ entry here to include these points: http://www.open-mpi.org/faq/?category=openfabrics#ofa-fork On Nov 30, 2010, at 9:52 AM, Ken Cain wrote: > Hi Jeff, > > We have had some recent experience with this in an Open MPI 1.4.x version and > thought it would be useful to contribute to the discussion. Please see below. > > Jeff Squyres wrote: >> On Nov 29, 2010, at 6:25 PM, George Bosilca wrote: >>> The main problem is that openib require to pin memory pages in order to >>> take advantage of RMA features. There is a major issues with these pinned >>> pages and fork, leading to segmentation fault in some specific cases. >>> However, we only pin the pages on the MPI calls related to data transfers. >>> Therefore, if you call fork __before__ any other MPI data transfer function >>> (but after MPI_Init as you use the process rank), your application should >>> be safe. >> Note that Open MPI also pins some internal memory during MPI_INIT, but that >> memory is totally internal to libmpi, so you should be safe (i.e., you >> should never be able to find it and therefore never be able to try to touch >> it). > > This is what we believe happened in our testing: > > 1. MPI_init allocated and pinned down some memory. This memory was 64 byte > aligned and not page-aligned to 4096 bytes. So an allocation that ideally > should have resulted in 2 pages being pinned, actually had 3 pages pinned > with lots of unused memory on the 3rd page. > > 2. A child process created via popen tried to allocate some memory (perhaps a > byproduct of popen execution itself) and was allocated memory on that last > page with lots of unused memory. When the child tried to touch the > allocation, there was seg fault. > > We could reduce the probability of this happenning by changing the alignment > of MPI allocations to 4096 bytes. But since MPI allocations are not sized to > be multiple of page size, this isn't a foolproof method. > > One way (agreed not ideal) to avoid the potential seg fault is to set the MCA > parameter btl_openib_want_fork_suppoort = 0. But then you are "trusting" any > child processes to not intentionally or as a result of a bug, touch the > memory regions that have been registered/pinned by the parent. > How can one be sure that the disabling the warning is ok? Could you please elaborate on what makes forks vulnerable? May be that will guide the developers to make an informed decision on whether to disable them or find another alternative. >>> No way to know at 100%. Now for an elaborate answer: Once upon a time ... >>> The fork story is a long and boring one, we would all have preferred to >>> never heard about it (believe me). A quick and compressed version can be >>> found on the QLogic download page >>> (http://filedownloads.qlogic.com/files/driver/70277/release_QLogicIB-Basic_4400_Rev_A.html). >> That's a good summary. The issue is with OFED itself, not with Open MPI. >> Note, too, that calling popen() should also be safe (even though we'll warn >> about it -- our atfork hook has no way of knowing whether you're calling >> system, popen, or something else). > > Thanks, > > -Ken > -- > Ken Cain > Mercury Computer Systems, Inc. (http://www.mc.com) > > This message is intended only for the designated recipient(s) and may > contain confidential or proprietary information of Mercury Computer > Systems, Inc. This message is solely intended to facilitate business > discussions and does not constitute an express or implied offer to sell > or purchase any products, services, or support. Any commitments must be > made in writing and signed by duly authorized representatives of each > party. > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
Hi Jeff, We have had some recent experience with this in an Open MPI 1.4.x version and thought it would be useful to contribute to the discussion. Please see below. Jeff Squyres wrote: On Nov 29, 2010, at 6:25 PM, George Bosilca wrote: The main problem is that openib require to pin memory pages in order to take advantage of RMA features. There is a major issues with these pinned pages and fork, leading to segmentation fault in some specific cases. However, we only pin the pages on the MPI calls related to data transfers. Therefore, if you call fork __before__ any other MPI data transfer function (but after MPI_Init as you use the process rank), your application should be safe. Note that Open MPI also pins some internal memory during MPI_INIT, but that memory is totally internal to libmpi, so you should be safe (i.e., you should never be able to find it and therefore never be able to try to touch it). This is what we believe happened in our testing: 1. MPI_init allocated and pinned down some memory. This memory was 64 byte aligned and not page-aligned to 4096 bytes. So an allocation that ideally should have resulted in 2 pages being pinned, actually had 3 pages pinned with lots of unused memory on the 3rd page. 2. A child process created via popen tried to allocate some memory (perhaps a byproduct of popen execution itself) and was allocated memory on that last page with lots of unused memory. When the child tried to touch the allocation, there was seg fault. We could reduce the probability of this happenning by changing the alignment of MPI allocations to 4096 bytes. But since MPI allocations are not sized to be multiple of page size, this isn't a foolproof method. One way (agreed not ideal) to avoid the potential seg fault is to set the MCA parameter btl_openib_want_fork_suppoort = 0. But then you are "trusting" any child processes to not intentionally or as a result of a bug, touch the memory regions that have been registered/pinned by the parent. How can one be sure that the disabling the warning is ok? Could you please elaborate on what makes forks vulnerable? May be that will guide the developers to make an informed decision on whether to disable them or find another alternative. No way to know at 100%. Now for an elaborate answer: Once upon a time ... The fork story is a long and boring one, we would all have preferred to never heard about it (believe me). A quick and compressed version can be found on the QLogic download page (http://filedownloads.qlogic.com/files/driver/70277/release_QLogicIB-Basic_4400_Rev_A.html). That's a good summary. The issue is with OFED itself, not with Open MPI. Note, too, that calling popen() should also be safe (even though we'll warn about it -- our atfork hook has no way of knowing whether you're calling system, popen, or something else). Thanks, -Ken -- Ken Cain Mercury Computer Systems, Inc. (http://www.mc.com) This message is intended only for the designated recipient(s) and may contain confidential or proprietary information of Mercury Computer Systems, Inc. This message is solely intended to facilitate business discussions and does not constitute an express or implied offer to sell or purchase any products, services, or support. Any commitments must be made in writing and signed by duly authorized representatives of each party.
Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
On Nov 30 2010, Ralph Castain wrote: Here is what one IB vendor says about the issue on their web site (redacted to protect the innocent): "At the time of this release, the (redacted-openib) driver has issues with buffers sharing pages when fork( ) is used. Pinned (locked in memory) pages are normally marked copy-on-write during a fork. That is TRULY demented! It is almost always precisely the wrong thing to do. If a page is pinned before a fork and subsequently written to while RDMA operations are being performed on the same page, silent data corruption can occur as RDMA operations continue to stream data to a page that has moved. To avoid this, the (redacted-openib) driver does not use copy-on-write behavior during a fork for pinned pages. Instead, access to these pages by the child process will result in a segmentation violation." That is sane. Not user-friendly, but at least sane. While there is some variation, I believe you will find that all IB comm shares this problem. So it is wise to avoid using fork if you want to use the openib transport. Yes and no. Some such communication may allow RDMA only to shared memory, which solves the problem in another way. Several specialist HPC networks were (are?) like that, and I can see no reason why an IB driver should not use the same design. That, of course, means that most MPI transfers need a copy. Hence the warning. Ignoring it is purely a "user beware" situation, but we provide that mechanism for the truly adventurous...or IB developers who want to someday resolve the problem. Well, there is a much simpler case where it will "just work", which is very probably what the OP was doing. When the fork is immediately followed by an exec in the child process, there isn't an issue. We all know the history, but the mainframe designs of having a proper spawn primitive were much cleaner. However, that's not what we've got. It might be worth adding to the note that this is the ONLY case when the ordinary user is advised to use that facility. Or it might not, depending on the level of Clue that readers are expected to have. Regards, Nick Maclaren.
Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
On Nov 29, 2010, at 6:25 PM, George Bosilca wrote: > The main problem is that openib require to pin memory pages in order to take > advantage of RMA features. There is a major issues with these pinned pages > and fork, leading to segmentation fault in some specific cases. However, we > only pin the pages on the MPI calls related to data transfers. Therefore, if > you call fork __before__ any other MPI data transfer function (but after > MPI_Init as you use the process rank), your application should be safe. Note that Open MPI also pins some internal memory during MPI_INIT, but that memory is totally internal to libmpi, so you should be safe (i.e., you should never be able to find it and therefore never be able to try to touch it). >> How can one be sure that the disabling the warning is ok? Could you please >> elaborate on what makes forks vulnerable? May be that will guide the >> developers to make an informed decision on whether to disable them or find >> another alternative. > > No way to know at 100%. Now for an elaborate answer: Once upon a time ... The > fork story is a long and boring one, we would all have preferred to never > heard about it (believe me). A quick and compressed version can be found on > the QLogic download page > (http://filedownloads.qlogic.com/files/driver/70277/release_QLogicIB-Basic_4400_Rev_A.html). That's a good summary. The issue is with OFED itself, not with Open MPI. Note, too, that calling popen() should also be safe (even though we'll warn about it -- our atfork hook has no way of knowing whether you're calling system, popen, or something else). -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
Here is what one IB vendor says about the issue on their web site (redacted to protect the innocent): "At the time of this release, the (redacted-openib) driver has issues with buffers sharing pages when fork( ) is used. Pinned (locked in memory) pages are normally marked copy-on-write during a fork. If a page is pinned before a fork and subsequently written to while RDMA operations are being performed on the same page, silent data corruption can occur as RDMA operations continue to stream data to a page that has moved. To avoid this, the (redacted-openib) driver does not use copy-on-write behavior during a fork for pinned pages. Instead, access to these pages by the child process will result in a segmentation violation." While there is some variation, I believe you will find that all IB comm shares this problem. So it is wise to avoid using fork if you want to use the openib transport. Hence the warning. Ignoring it is purely a "user beware" situation, but we provide that mechanism for the truly adventurous...or IB developers who want to someday resolve the problem. On Nov 29, 2010, at 3:44 PM, wrote: > George > > Thanks for the explanation. I am trying to understand the following line in > your mail: > > “In fact, any fork done prior to the communication is a non-issue, but it is > difficult to identify. Therefore, we output the warning as soon as we detect > a fork after MPI_Init.” > > Does it mean that if I have a fork() after the communication (ie; mpi_send or > mpi_receive etc), I may have to relook at a different implementation to be at > safe side? I don’t want to suppress the messages if they result in any > corruption later. > > How can one be sure that the disabling the warning is ok? Could you please > elaborate on what makes forks vulnerable? May be that will guide the > developers to make an informed decision on whether to disable them or find > another alternative. > > > > Thanks > > Ananda > > ---------- PREVIOUS MESSAGE ------------- > > Subject: Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!! > From: George Bosilca (bosilca_at_[hidden]) > Date: 2010-11-29 12:22:15 > > Next message: Jeff Squyres: "Re: [OMPI devel] Warning on fork() disappears if > I use MPI threads!!" > Previous message: ananda.mudar_at_[hidden]: "Re: [OMPI devel] Warning on > fork() disappears if I use MPI threads!!" > In reply to: ananda.mudar_at_[hidden]: "Re: [OMPI devel] Warning on fork() > disappears if I use MPI threads!!" > Next in thread: N.M. Maclaren: "Re: [OMPI devel] Warning on fork() disappears > if I use MPI threads!!" > Reply: N.M. Maclaren: "Re: [OMPI devel] Warning on fork() disappears if I use > MPI threads!!" > If your code doesn't exactly what is described in the code snippet attached > to your previous email, then you can safely ignore the warning. In fact, any > fork done prior to the communication is a non-issue, but it is difficult to > identify. Therefore, we output the warning as soon as we detect a fork after > MPI_Init. > > You can find more information about the usage of fork in Open MPI at > http://www.open-mpi.de/faq/?category=tuning#fork-warning > > george. > > On Nov 29, 2010, at 12:12 , wrote: > > > I am posting this question again as it was sent before the long weekend and > > didn’t see any responses so far. Can anyone please explain the discrepancy > > I am observing with the scenario explained in the post below? > > > > Thanks > > Ananda > > Sent: Tuesday, November 23, 2010 2:24 PM > > To: devel_at_[hidden] > > Subject: Warning on fork() disappears if I use MPI threads!! > > > > Hi > > > > I am running into a very wierd problem. > > > > If I initialize MPI normally ie; with MPI_Init(), and make one of the MPI > > process to do "popen()" call, I get the following warning/error message: > > > > == Message start === > > An MPI process has executed an operation involving a call to the > > "fork()" system call to create a child process. Open MPI is currently > > operating in a condition that could result in memory corruption or > > other system errors; your MPI job may hang, crash, or produce silent > > data corruption. The use of fork() (or system() or other calls that > > create child processes) is strongly discouraged. > > == Message end > > > > However this error message goes away, if I initialize MPI with threads ie; > > MPI_Init_thread(). Can anyone explain this discrepancy? > > > > I am giving a snippet of th
Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
On Nov 29, 2010, at 17:44 , wrote: > George > > Thanks for the explanation. I am trying to understand the following line in > your mail: > > “In fact, any fork done prior to the communication is a non-issue, but it is > difficult to identify. Therefore, we output the warning as soon as we detect > a fork after MPI_Init.” > > Does it mean that if I have a fork() after the communication (ie; mpi_send or > mpi_receive etc), I may have to relook at a different implementation to be at > safe side? I don’t want to suppress the messages if they result in any > corruption later. The main problem is that openib require to pin memory pages in order to take advantage of RMA features. There is a major issues with these pinned pages and fork, leading to segmentation fault in some specific cases. However, we only pin the pages on the MPI calls related to data transfers. Therefore, if you call fork __before__ any other MPI data transfer function (but after MPI_Init as you use the process rank), your application should be safe. > How can one be sure that the disabling the warning is ok? Could you please > elaborate on what makes forks vulnerable? May be that will guide the > developers to make an informed decision on whether to disable them or find > another alternative. No way to know at 100%. Now for an elaborate answer: Once upon a time ... The fork story is a long and boring one, we would all have preferred to never heard about it (believe me). A quick and compressed version can be found on the QLogic download page (http://filedownloads.qlogic.com/files/driver/70277/release_QLogicIB-Basic_4400_Rev_A.html). > Arbitrary fork() support is not supported in the OpenFabrics software stack. > If you use fork() in your application, you must not touch any registered > memory before calling some form of exec() to launch another process. Calling > system() is safe. This limitation includes both native InfiniBand > applications and MPI applications using a RDMA based implementation of MPI > (such as OFED's openmpi and mvapich). george. > > > > Thanks > > Ananda > > ---------- PREVIOUS MESSAGE ------------- > > Subject: Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!! > From: George Bosilca (bosilca_at_[hidden]) > Date: 2010-11-29 12:22:15 > > • Next message: Jeff Squyres: "Re: [OMPI devel] Warning on fork() > disappears if I use MPI threads!!" > • Previous message: ananda.mudar_at_[hidden]: "Re: [OMPI devel] Warning > on fork() disappears if I use MPI threads!!" > • In reply to: ananda.mudar_at_[hidden]: "Re: [OMPI devel] Warning on > fork() disappears if I use MPI threads!!" > • Next in thread: N.M. Maclaren: "Re: [OMPI devel] Warning on fork() > disappears if I use MPI threads!!" > • Reply: N.M. Maclaren: "Re: [OMPI devel] Warning on fork() disappears > if I use MPI threads!!" > If your code doesn't exactly what is described in the code snippet attached > to your previous email, then you can safely ignore the warning. In fact, any > fork done prior to the communication is a non-issue, but it is difficult to > identify. Therefore, we output the warning as soon as we detect a fork after > MPI_Init. > > You can find more information about the usage of fork in Open MPI at > http://www.open-mpi.de/faq/?category=tuning#fork-warning > > george. > > On Nov 29, 2010, at 12:12 , wrote: > > > I am posting this question again as it was sent before the long weekend and > > didn’t see any responses so far. Can anyone please explain the discrepancy > > I am observing with the scenario explained in the post below? > > > > Thanks > > Ananda > > Sent: Tuesday, November 23, 2010 2:24 PM > > To: devel_at_[hidden] > > Subject: Warning on fork() disappears if I use MPI threads!! > > > > Hi > > > > I am running into a very wierd problem. > > > > If I initialize MPI normally ie; with MPI_Init(), and make one of the MPI > > process to do "popen()" call, I get the following warning/error message: > > > > == Message start === > > An MPI process has executed an operation involving a call to the > > "fork()" system call to create a child process. Open MPI is currently > > operating in a condition that could result in memory corruption or > > other system errors; your MPI job may hang, crash, or produce silent > > data corruption. The use of fork() (or system() or other calls that > > create child processes) is strongly discouraged. > > == Message end > > > > However this e
Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
George Thanks for the explanation. I am trying to understand the following line in your mail: "In fact, any fork done prior to the communication is a non-issue, but it is difficult to identify. Therefore, we output the warning as soon as we detect a fork after MPI_Init." Does it mean that if I have a fork() after the communication (ie; mpi_send or mpi_receive etc), I may have to relook at a different implementation to be at safe side? I don't want to suppress the messages if they result in any corruption later. How can one be sure that the disabling the warning is ok? Could you please elaborate on what makes forks vulnerable? May be that will guide the developers to make an informed decision on whether to disable them or find another alternative. Thanks Ananda -- PREVIOUS MESSAGE ----- Subject: Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!! From: George Bosilca (bosilca_at_[hidden]) List-Post: devel@lists.open-mpi.org Date: 2010-11-29 12:22:15 * Next message: Jeff Squyres: "Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!" <http://www.open-mpi.org/community/lists/devel/2010/11/8730.php> * Previous message: ananda.mudar_at_[hidden]: "Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!" <http://www.open-mpi.org/community/lists/devel/2010/11/8728.php> * In reply to: ananda.mudar_at_[hidden]: "Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!" <http://www.open-mpi.org/community/lists/devel/2010/11/8728.php> * Next in thread: N.M. Maclaren: "Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!" <http://www.open-mpi.org/community/lists/devel/2010/11/8731.php> * Reply: N.M. Maclaren: "Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!" <http://www.open-mpi.org/community/lists/devel/2010/11/8731.php> If your code doesn't exactly what is described in the code snippet attached to your previous email, then you can safely ignore the warning. In fact, any fork done prior to the communication is a non-issue, but it is difficult to identify. Therefore, we output the warning as soon as we detect a fork after MPI_Init. You can find more information about the usage of fork in Open MPI at http://www.open-mpi.de/faq/?category=tuning#fork-warning george. On Nov 29, 2010, at 12:12 , wrote: > I am posting this question again as it was sent before the long weekend and didn't see any responses so far. Can anyone please explain the discrepancy I am observing with the scenario explained in the post below? > > Thanks > Ananda > Sent: Tuesday, November 23, 2010 2:24 PM > To: devel_at_[hidden] > Subject: Warning on fork() disappears if I use MPI threads!! > > Hi > > I am running into a very wierd problem. > > If I initialize MPI normally ie; with MPI_Init(), and make one of the MPI process to do "popen()" call, I get the following warning/error message: > > == Message start === > An MPI process has executed an operation involving a call to the > "fork()" system call to create a child process. Open MPI is currently > operating in a condition that could result in memory corruption or > other system errors; your MPI job may hang, crash, or produce silent > data corruption. The use of fork() (or system() or other calls that > create child processes) is strongly discouraged. > == Message end > > However this error message goes away, if I initialize MPI with threads ie; MPI_Init_thread(). Can anyone explain this discrepancy? > > I am giving a snippet of the program that causes this problem: > > == Code snippet start == > if ( rank == 0) { > output = popen("ls -l", "r"); > while((c=getc(output))!=EOF) > printf("%c",c); > pclose(output); > } > == Code snippet end == > > If this is a design constraint, how can I overcome this problem. > > Thanks > Ananda > > Ananda B Mudar, PMP > Senior Technical Architect > Wipro Technologies > Ph: 972 765 8093 begin_of_the_skype_highlighting 972 765 8093 end_of_the_skype_highlighting > Please do not print this email unless it is absolutely necessary. > > The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. > > WARNING: Computer viruses can be transmitted via email. The recipient should
Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
Jeff I am invoking MPI_Init_thread() with MPI_THREAD_MULTIPLE. If openib BTL is responsible for the warning and openib BTL is excluded in this case, then that explains the discrepancy. FYI, I invoked MPI_Init_thread() with MPI_THREAD_SERIALIZED and I got the warning message about the fork(). Thanks Ananda --- PREVIOUS MESSAGE - Subject: Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!! From: Jeff Squyres (jsquyres_at_[hidden]) List-Post: devel@lists.open-mpi.org Date: 2010-11-29 14:10:44 * Next message: N.M. Maclaren: "Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!" <http://www.open-mpi.org/community/lists/devel/2010/11/8731.php> * Previous message: George Bosilca: "Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!" <http://www.open-mpi.org/community/lists/devel/2010/11/8729.php> * In reply to: ananda.mudar_at_[hidden]: "[OMPI devel] Warning on fork() disappears if I use MPI threads!!" <http://www.open-mpi.org/community/lists/devel/2010/11/8698.php> On Nov 23, 2010, at 3:23 PM, wrote: > However this error message goes away, if I initialize MPI with threads ie; MPI_Init_thread(). Can anyone explain this discrepancy? What thread level are you invoking MPI_INIT_THREAD with? One possible reason this could be happening is that the openib BTL is excluding itself if you use MPI_THREAD_MULTIPLE. The openib BTL is the entity that is responsible for the fork warning -- if it's not being used, then no warning is issued because there is no problem with forking. -- Jeff Squyres jsquyres_at_[hidden] For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
On Nov 29 2010, George Bosilca wrote: If your code doesn't exactly what is described in the code snippet attached to your previous email, then you can safely ignore the warning. In fact, any fork done prior to the communication is a non-issue, but it is difficult to identify. Therefore, we output the warning as soon as we detect a fork after MPI_Init. USUALLY a non-issue. You can find more information about the usage of fork in Open MPI at http://www.open-mpi.de/faq/?category=tuning#fork-warning Yes. It says about as much as can be said, except between consenting geeks in private! Well, I suppose that this mailing list counts :-) I have had trouble with systems that sent signals to process groups and/or all processes with a particular controlling terminal, and even operating system 'super groups', where the location of the fork won't save you. I can't see how to describe that issue in terms that make sense to most users. Regards, Nick Maclaren.
Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
On Nov 23, 2010, at 3:23 PM, wrote: > However this error message goes away, if I initialize MPI with threads ie; > MPI_Init_thread(). Can anyone explain this discrepancy? What thread level are you invoking MPI_INIT_THREAD with? One possible reason this could be happening is that the openib BTL is excluding itself if you use MPI_THREAD_MULTIPLE. The openib BTL is the entity that is responsible for the fork warning -- if it's not being used, then no warning is issued because there is no problem with forking. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
If your code doesn't exactly what is described in the code snippet attached to your previous email, then you can safely ignore the warning. In fact, any fork done prior to the communication is a non-issue, but it is difficult to identify. Therefore, we output the warning as soon as we detect a fork after MPI_Init. You can find more information about the usage of fork in Open MPI at http://www.open-mpi.de/faq/?category=tuning#fork-warning george. On Nov 29, 2010, at 12:12 , wrote: > I am posting this question again as it was sent before the long weekend and > didn’t see any responses so far. Can anyone please explain the discrepancy I > am observing with the scenario explained in the post below? > > Thanks > Ananda > Sent: Tuesday, November 23, 2010 2:24 PM > To: de...@open-mpi.org > Subject: Warning on fork() disappears if I use MPI threads!! > > Hi > > I am running into a very wierd problem. > > If I initialize MPI normally ie; with MPI_Init(), and make one of the MPI > process to do "popen()" call, I get the following warning/error message: > > == Message start === > An MPI process has executed an operation involving a call to the > "fork()" system call to create a child process. Open MPI is currently > operating in a condition that could result in memory corruption or > other system errors; your MPI job may hang, crash, or produce silent > data corruption. The use of fork() (or system() or other calls that > create child processes) is strongly discouraged. > == Message end > > However this error message goes away, if I initialize MPI with threads ie; > MPI_Init_thread(). Can anyone explain this discrepancy? > > I am giving a snippet of the program that causes this problem: > > == Code snippet start == > if ( rank == 0) { > output = popen("ls -l", "r"); > while((c=getc(output))!=EOF) > printf("%c",c); > pclose(output); > } > == Code snippet end == > > If this is a design constraint, how can I overcome this problem. > > Thanks > Ananda > > Ananda B Mudar, PMP > Senior Technical Architect > Wipro Technologies > Ph: 972 765 8093 > Please do not print this email unless it is absolutely necessary. > > The information contained in this electronic message and any attachments to > this message are intended for the exclusive use of the addressee(s) and may > contain proprietary, confidential or privileged information. If you are not > the intended recipient, you should not disseminate, distribute or copy this > e-mail. Please notify the sender immediately and destroy all copies of this > message and any attachments. > > WARNING: Computer viruses can be transmitted via email. The recipient should > check this email and any attachments for the presence of viruses. The company > accepts no liability for any damage caused by any virus transmitted by this > email. > > www.wipro.com > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] Warning on fork() disappears if I use MPI threads!!
I am posting this question again as it was sent before the long weekend and didn't see any responses so far. Can anyone please explain the discrepancy I am observing with the scenario explained in the post below? Thanks Ananda Sent: Tuesday, November 23, 2010 2:24 PM To: de...@open-mpi.org Subject: Warning on fork() disappears if I use MPI threads!! Hi I am running into a very wierd problem. If I initialize MPI normally ie; with MPI_Init(), and make one of the MPI process to do "popen()" call, I get the following warning/error message: == Message start === An MPI process has executed an operation involving a call to the "fork()" system call to create a child process. Open MPI is currently operating in a condition that could result in memory corruption or other system errors; your MPI job may hang, crash, or produce silent data corruption. The use of fork() (or system() or other calls that create child processes) is strongly discouraged. == Message end However this error message goes away, if I initialize MPI with threads ie; MPI_Init_thread(). Can anyone explain this discrepancy? I am giving a snippet of the program that causes this problem: == Code snippet start == if ( rank == 0) { output = popen("ls -l", "r"); while((c=getc(output))!=EOF) printf("%c",c); pclose(output); } == Code snippet end == If this is a design constraint, how can I overcome this problem. Thanks Ananda Ananda B Mudar, PMP Senior Technical Architect Wipro Technologies Ph: 972 765 8093 Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com