Dear Meep users ! Lately I was trying to implement pythonic interface for libmeep. My current goal is enabling mpi feature of libmeep in python scripts. While enabling mpi is as straight as just calling mpi_init with correct arguments, the actual mpi acceleration do not meet my expectations. I did some tests with meep-mpi and c++ tasks also and found that the results are essentially the same - mpi efficiency is lower than expected. For mpich the overall calculation speed on dual-core pentium was actually slower than on single core (mpi interconnection took way too much time). With openmpi, things got better. My simple tests, performed with meep-mpi (scheme), c++ compiled testfiles (available in meep source) and pythonic interface all show approx 20-30% acceleraton (at best, 40% in time stepping code, see below) comparing dual-core pentium to single-core on the same PC. My expectation for such SMP system was much higher - about 75-80%
So the first question is - what mpi efficiency do you observe on your systems? Can someone provide simple benchmark (ctl preferred), which do not take hours to run on desktop PC and clearly demonstrates advantage of mpi? Next question - there is a special argument - num_chunks in class structure. Is it supposed to be the number of available processor cores, so that calculation domain optimally split among nodes ? And the last - Are there any hints for using meep with mpi support ? With best regards Nizamov Shawkat Supplementary: > cat 1.ctl (set! geometry-lattice (make lattice (size 16 8 no-size))) (set! geometry (list (make block (center 0 0) (size infinity 1 infinity) (material (make dielectric (epsilon 12)))))) (set! sources (list (make source (src (make continuous-src (frequency 0.15))) (component Ez) (center -7 0)))) (set! pml-layers (list (make pml (thickness 1.0)))) (set! resolution 10) (run-until 2000 (at-beginning output-epsilon) (at-end output-efield-z)) > mpirun -np 1 /usr/bin/meep-mpi 1.ctl Using MPI version 2.0, 1 processes ----------- Initializing structure... Working in 2D dimensions. block, center = (0,0,0) size (1e+20,1,1e+20) axes (1,0,0), (0,1,0), (0,0,1) dielectric constant epsilon = 12 time for set_epsilon = 0.054827 s ----------- creating output file "./1-eps-000000.00.h5"... Meep progress: 230.95/2000.0 = 11.5% done in 4.0s, 30.6s to go on time step 4625 (time=231.25), 0.000865026 s/step Meep progress: 468.55/2000.0 = 23.4% done in 8.0s, 26.2s to go on time step 9378 (time=468.9), 0.000841727 s/step Meep progress: 705.8/2000.0 = 35.3% done in 12.0s, 22.0s to go on time step 14123 (time=706.15), 0.000843144 s/step Meep progress: 943.35/2000.0 = 47.2% done in 16.0s, 17.9s to go on time step 18874 (time=943.7), 0.000841985 s/step Meep progress: 1181.4/2000.0 = 59.1% done in 20.0s, 13.9s to go on time step 23635 (time=1181.75), 0.00084028 s/step Meep progress: 1418.85/2000.0 = 70.9% done in 24.0s, 9.8s to go on time step 28384 (time=1419.2), 0.000842386 s/step Meep progress: 1654.05/2000.0 = 82.7% done in 28.0s, 5.9s to go on time step 33088 (time=1654.4), 0.000850374 s/step Meep progress: 1891.5/2000.0 = 94.6% done in 32.0s, 1.8s to go on time step 37837 (time=1891.85), 0.000842369 s/step creating output file "./1-ez-002000.00.h5"... run 0 finished at t = 2000.0 (40000 timesteps) Elapsed run time = 33.9869 s > mpirun -np 2 /usr/bin/meep-mpi 1.ctl Using MPI version 2.0, 2 processes ----------- Initializing structure... Working in 2D dimensions. block, center = (0,0,0) size (1e+20,1,1e+20) axes (1,0,0), (0,1,0), (0,0,1) dielectric constant epsilon = 12 time for set_epsilon = 0.0299381 s ----------- creating output file "./1-eps-000000.00.h5"... Meep progress: 328.85/2000.0 = 16.4% done in 4.0s, 20.4s to go on time step 6577 (time=328.85), 0.00060946 s/step Meep progress: 656.1/2000.0 = 32.8% done in 8.0s, 16.4s to go on time step 13123 (time=656.15), 0.000611187 s/step Meep progress: 985.9/2000.0 = 49.3% done in 12.0s, 12.4s to go on time step 19719 (time=985.95), 0.000606462 s/step Meep progress: 1315.1/2000.0 = 65.8% done in 16.0s, 8.3s to go on time step 26302 (time=1315.1), 0.000608525 s/step Meep progress: 1644.95/2000.0 = 82.2% done in 20.0s, 4.3s to go on time step 32911 (time=1645.55), 0.000605277 s/step Meep progress: 1975.0/2000.0 = 98.8% done in 24.0s, 0.3s to go on time step 39512 (time=1975.6), 0.000606022 s/step creating output file "./1-ez-002000.00.h5"... run 0 finished at t = 2000.0 (40000 timesteps) Elapsed run time = 24.57 s For python script: (skipped harmless warnings like [ubuntu:24234] mca: base: component_find: unable to open osc pt2pt: file not found (ignored)) > mpirun -np 1 ./test-tut1-mpi.py Using MPI version 2.0, 1 processes Count processors: 1 My rank is 0 time for set_epsilon = 0.696046 s creating output file "./eps-000000.00.h5"... on time step 1880 (time=94), 0.00212839 s/step on time step 3781 (time=189.05), 0.00210516 s/step on time step 5682 (time=284.1), 0.00210487 s/step on time step 7585 (time=379.25), 0.00210279 s/step on time step 9486 (time=474.3), 0.00210501 s/step Field time usage: connnecting chunks: 0.0260548 s time stepping: 17.3853 s communicating: 3.30172 s outputting fields: 0.0123442 s Fourier transforming: 0.0178975 s everything else: 0.362632 s > mpirun -np 2 ./test-tut1-mpi.py Using MPI version 2.0, 2 processes Count processors: 2 My rank is 0 time for set_epsilon = 0.37543 s creating output file "./eps-000000.00.h5"... on time step 2293 (time=114.65), 0.00174445 s/step on time step 4558 (time=227.9), 0.00176638 s/step on time step 6874 (time=343.7), 0.00172781 s/step on time step 9174 (time=458.7), 0.00173953 s/step Field time usage: connnecting chunks: 0.0244552 s time stepping: 12.9744 s communicating: 4.15587 s outputting fields: 0.00916502 s Fourier transforming: 0.0171199 s everything else: 0.328006 s > mpirun -np 2 ./test-tut1-mpi.py (class structure created with num_chunks=2) Using MPI version 2.0, 2 processes Count processors: 2 My rank is 0 time for set_epsilon = 0.365422 s creating output file "./eps-000000.00.h5"... on time step 2066 (time=103.3), 0.00193695 s/step on time step 4152 (time=207.6), 0.00191805 s/step on time step 6205 (time=310.25), 0.00194897 s/step on time step 8307 (time=415.35), 0.00190318 s/step Field time usage: connnecting chunks: 0.024951 s time stepping: 14.0125 s communicating: 4.72859 s outputting fields: 0.00907009 s Fourier transforming: 0.0204064 s everything else: 0.438375 s
_______________________________________________ meep-discuss mailing list meep-discuss@ab-initio.mit.edu http://ab-initio.mit.edu/cgi-bin/mailman/listinfo/meep-discuss