Re: [deal.II] Cluster with Infiniband network
On 11/17/2016 05:41 AM, Denis Davydov wrote: On Wednesday, November 16, 2016 at 3:11:08 PM UTC+1, Wolfgang Bangerth wrote: Our rule of thumb is usually to use about 100,000 unknowns per MPI process. @Woflgang: maybe this should be added to a stand-along page in https://www.dealii.org/developer/index.html within the "Information for users" which summarises best practices using deal.II on clusters. This number is actually already in the documentation: in the glossary entry on the topic of scalability. This entry is also linked to from the distributed computing documentation module, pretty far at the top. I'm not opposed to linking to it from elsewhere. The issue is simply that few places actually come to mind where this makes sense. It doesn't help if we add such a reference to obscure places where nobody goes looking for it. I will, however, add a couple of comments to the results section of step-40. That is the first parallel programming program most people see. Best W. -- Wolfgang Bangerth email: bange...@colostate.edu www: http://www.math.colostate.edu/~bangerth/ -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to dealii+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [deal.II] Cluster with Infiniband network
On 11/17/2016 06:40 AM, Ali Dorostkar wrote: Assume that we have 1,000,000 DOFs per compute node which means 1,000,000 DOFs per MPI process for one MPI process this means roughly 10GB of memory. Now if we use 10 core per node, it means 100,000 DOFs per MPI process and the amount of memory per MPI process should become 1GB. In any case the amount of memory per compute node should still be 10GB. This is the reason I don’t follow why the memory usage should change so drastically by adding the number of cores per compute nodes. Oh, I see -- when you say 2 MPI processes per node, you also doubled the overall number of MPI processes, not just packed the same number of processes onto half the number of nodes. Either way, see if you can monitor how much memory you are using and if you're simply getting close to or running over the amount of memory your nodes have. Best W. -- Wolfgang Bangerth email: bange...@colostate.edu www: http://www.math.colostate.edu/~bangerth/ -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to dealii+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [deal.II] Cluster with Infiniband network
On Wednesday, November 16, 2016 at 3:11:08 PM UTC+1, Wolfgang Bangerth wrote: > > Our rule of thumb is usually to use about 100,000 > unknowns per MPI process. @Woflgang: maybe this should be added to a stand-along page in https://www.dealii.org/developer/index.html within the "Information for users" which summarises best practices using deal.II on clusters. Regards, Denis. -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to dealii+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [deal.II] Cluster with Infiniband network
On 11/16/2016 04:31 AM, Ashkan Dorostkar wrote: Could you elaborate on why I am using twice more memory on each node if I use more than one core per node? If you use two cores per node, then you are running two programs within the same amount of memory, each of which will allocate their own data structures etc. I have tried using around 300 nodes which means roughly one million DOFs per node but still it fails. This is the reason I don’t think the problem is related to memory. This is a lot of unknowns. Depending on your application, you need to expect 10 kB of memory per unknown, sometimes even more. So trying to solve 1 million unknowns per node is a lot. Our rule of thumb is usually to use about 100,000 unknowns per MPI process. Best W. -- Wolfgang Bangerth email: bange...@colostate.edu www: http://www.math.colostate.edu/~bangerth/ -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to dealii+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [deal.II] Cluster with Infiniband network
Hi again, Sorry for the late reply. Believe it or not, the internet connection is terrible here. The failure happens when the number of degree of freedoms is 96 million or higher. This happens every time I run the program. I have has this issue on another cluster but I managed to avoid it by using the so-called fat nodes which were providing 128GB of ram. Here all nodes have 64GB ram. The interesting thing is that if I used one core on each node then the program runs successfully but if I use more than one core, I see failure. I don't think this is due to any lose cable, as for MPI, I don't know how to test other settings. On Wednesday, November 9, 2016 at 9:29:09 PM UTC+3, Wolfgang Bangerth wrote: > > On 11/09/2016 08:36 AM, Ashkan Dorostkar wrote: > > > > [n49422:9059] *** An error occurred in MPI_Allreduce > > [n49422:9059] *** reported by process [3040346113,140733193388063] > > [n49422:9059] *** on communicator MPI_COMM_WORLD > > [n49422:9059] *** MPI_ERR_IN_STATUS: error code in status > > [n49422:9059] *** MPI_ERRORS_ARE_FATAL (processes in this communicator > > will now abort, > > [n49422:9059] ***and potentially your MPI job) > > > > This looks like a communication issue or a communication oversaturation. > > Does anyone have any experience with this? > > Ashkan, > we have had people report such issues before, on an intermittent basis. > It is of course not impossible that this points to an actual problem in > deal.II (or PETSc, or Trilinos), but it is difficult to know for sure > without a backtrace where this came from. > > Does it happen every time with the same program? If it doesn't, it is > also possible that this is a symptom of a lose cable or a wrong > configuration of the MPI system -- both things we have seen in cluster > in the past. > > Best > W. > > -- > > Wolfgang Bangerth email: bang...@colostate.edu > www: http://www.math.colostate.edu/~bangerth/ > -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to dealii+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [deal.II] Cluster with Infiniband network
On 11/09/2016 08:36 AM, Ashkan Dorostkar wrote: [n49422:9059] *** An error occurred in MPI_Allreduce [n49422:9059] *** reported by process [3040346113,140733193388063] [n49422:9059] *** on communicator MPI_COMM_WORLD [n49422:9059] *** MPI_ERR_IN_STATUS: error code in status [n49422:9059] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [n49422:9059] ***and potentially your MPI job) This looks like a communication issue or a communication oversaturation. Does anyone have any experience with this? Ashkan, we have had people report such issues before, on an intermittent basis. It is of course not impossible that this points to an actual problem in deal.II (or PETSc, or Trilinos), but it is difficult to know for sure without a backtrace where this came from. Does it happen every time with the same program? If it doesn't, it is also possible that this is a symptom of a lose cable or a wrong configuration of the MPI system -- both things we have seen in cluster in the past. Best W. -- Wolfgang Bangerth email: bange...@colostate.edu www: http://www.math.colostate.edu/~bangerth/ -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to dealii+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[deal.II] Cluster with Infiniband network
Hello all, I am running a simulation of linear elasticity in the Lomonosov2 cluster (rank 41 in top 500) which has InfiniBand network. For problems larger than 90 million unknowns openmpi just aborts the program with this message [n49422:9059] *** An error occurred in MPI_Allreduce [n49422:9059] *** reported by process [3040346113,140733193388063] [n49422:9059] *** on communicator MPI_COMM_WORLD [n49422:9059] *** MPI_ERR_IN_STATUS: error code in status [n49422:9059] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [n49422:9059] ***and potentially your MPI job) This looks like a communication issue or a communication oversaturation. Does anyone have any experience with this? Best, Ashkan P.S I use the following flag to avoid this issue but it doesn't work either export OMPI_MCA_btl_openib_receive_queues="X,256,1024:X,65536,128" -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to dealii+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
