Dear Meep users !
Lately I was trying to implement pythonic interface for libmeep. My current
goal is enabling mpi feature of libmeep in python scripts. While enabling
mpi is as straight as just calling mpi_init with correct arguments, the
actual mpi acceleration do not meet my expectations. I did some tests with
meep-mpi and c++ tasks also and found that the results are essentially the
same - mpi efficiency is lower than expected. For mpich the overall
calculation speed on dual-core pentium was actually slower than on single
core (mpi interconnection took way too much time). With openmpi, things got
better. My simple tests, performed with meep-mpi (scheme), c++ compiled
testfiles (available in meep source) and pythonic interface all show approx
20-30% acceleraton (at best, 40% in time stepping code, see below) comparing
dual-core pentium to single-core on the same PC. My expectation for such
SMP system was much higher - about 75-80%
So the first question is - what mpi efficiency do you observe on your
systems? Can someone provide simple benchmark (ctl preferred), which do not
take hours to run on desktop PC and clearly demonstrates advantage of mpi?
Next question - there is a special argument - num_chunks in class structure.
Is it supposed to be the number of available processor cores, so that
calculation domain optimally split among nodes ?
And the last - Are there any hints for using meep with mpi support ?
With best regards
Nizamov Shawkat
Supplementary:
> cat 1.ctl
(set! geometry-lattice (make lattice (size 16 8 no-size)))
(set! geometry (list
(make block (center 0 0) (size infinity 1 infinity)
(material (make dielectric (epsilon 12))))))
(set! sources (list
(make source
(src (make continuous-src (frequency 0.15)))
(component Ez)
(center -7 0))))
(set! pml-layers (list (make pml (thickness 1.0))))
(set! resolution 10)
(run-until 2000
(at-beginning output-epsilon)
(at-end output-efield-z))
> mpirun -np 1 /usr/bin/meep-mpi 1.ctl
Using MPI version 2.0, 1 processes
-----------
Initializing structure...
Working in 2D dimensions.
block, center = (0,0,0)
size (1e+20,1,1e+20)
axes (1,0,0), (0,1,0), (0,0,1)
dielectric constant epsilon = 12
time for set_epsilon = 0.054827 s
-----------
creating output file "./1-eps-000000.00.h5"...
Meep progress: 230.95/2000.0 = 11.5% done in 4.0s, 30.6s to go
on time step 4625 (time=231.25), 0.000865026 s/step
Meep progress: 468.55/2000.0 = 23.4% done in 8.0s, 26.2s to go
on time step 9378 (time=468.9), 0.000841727 s/step
Meep progress: 705.8/2000.0 = 35.3% done in 12.0s, 22.0s to go
on time step 14123 (time=706.15), 0.000843144 s/step
Meep progress: 943.35/2000.0 = 47.2% done in 16.0s, 17.9s to go
on time step 18874 (time=943.7), 0.000841985 s/step
Meep progress: 1181.4/2000.0 = 59.1% done in 20.0s, 13.9s to go
on time step 23635 (time=1181.75), 0.00084028 s/step
Meep progress: 1418.85/2000.0 = 70.9% done in 24.0s, 9.8s to go
on time step 28384 (time=1419.2), 0.000842386 s/step
Meep progress: 1654.05/2000.0 = 82.7% done in 28.0s, 5.9s to go
on time step 33088 (time=1654.4), 0.000850374 s/step
Meep progress: 1891.5/2000.0 = 94.6% done in 32.0s, 1.8s to go
on time step 37837 (time=1891.85), 0.000842369 s/step
creating output file "./1-ez-002000.00.h5"...
run 0 finished at t = 2000.0 (40000 timesteps)
Elapsed run time = 33.9869 s
> mpirun -np 2 /usr/bin/meep-mpi 1.ctl
Using MPI version 2.0, 2 processes
-----------
Initializing structure...
Working in 2D dimensions.
block, center = (0,0,0)
size (1e+20,1,1e+20)
axes (1,0,0), (0,1,0), (0,0,1)
dielectric constant epsilon = 12
time for set_epsilon = 0.0299381 s
-----------
creating output file "./1-eps-000000.00.h5"...
Meep progress: 328.85/2000.0 = 16.4% done in 4.0s, 20.4s to go
on time step 6577 (time=328.85), 0.00060946 s/step
Meep progress: 656.1/2000.0 = 32.8% done in 8.0s, 16.4s to go
on time step 13123 (time=656.15), 0.000611187 s/step
Meep progress: 985.9/2000.0 = 49.3% done in 12.0s, 12.4s to go
on time step 19719 (time=985.95), 0.000606462 s/step
Meep progress: 1315.1/2000.0 = 65.8% done in 16.0s, 8.3s to go
on time step 26302 (time=1315.1), 0.000608525 s/step
Meep progress: 1644.95/2000.0 = 82.2% done in 20.0s, 4.3s to go
on time step 32911 (time=1645.55), 0.000605277 s/step
Meep progress: 1975.0/2000.0 = 98.8% done in 24.0s, 0.3s to go
on time step 39512 (time=1975.6), 0.000606022 s/step
creating output file "./1-ez-002000.00.h5"...
run 0 finished at t = 2000.0 (40000 timesteps)
Elapsed run time = 24.57 s
For python script:
(skipped harmless warnings like [ubuntu:24234] mca: base: component_find:
unable to open osc pt2pt: file not found (ignored))
> mpirun -np 1 ./test-tut1-mpi.py
Using MPI version 2.0, 1
processes
Count processors: 1 My rank is
0
time for set_epsilon = 0.696046
s
creating output file
"./eps-000000.00.h5"...
on time step 1880 (time=94), 0.00212839
s/step
on time step 3781 (time=189.05), 0.00210516
s/step
on time step 5682 (time=284.1), 0.00210487
s/step
on time step 7585 (time=379.25), 0.00210279
s/step
on time step 9486 (time=474.3), 0.00210501
s/step
Field time usage:
connnecting chunks: 0.0260548 s
time stepping: 17.3853 s
communicating: 3.30172 s
outputting fields: 0.0123442 s
Fourier transforming: 0.0178975 s
everything else: 0.362632 s
> mpirun -np 2 ./test-tut1-mpi.py
Using MPI version 2.0, 2
processes
Count processors: 2 My rank is
0
time for set_epsilon = 0.37543
s
creating output file
"./eps-000000.00.h5"...
on time step 2293 (time=114.65), 0.00174445
s/step
on time step 4558 (time=227.9), 0.00176638
s/step
on time step 6874 (time=343.7), 0.00172781
s/step
on time step 9174 (time=458.7), 0.00173953
s/step
Field time usage:
connnecting chunks: 0.0244552 s
time stepping: 12.9744 s
communicating: 4.15587 s
outputting fields: 0.00916502 s
Fourier transforming: 0.0171199 s
everything else: 0.328006 s
> mpirun -np 2 ./test-tut1-mpi.py
(class structure created with num_chunks=2)
Using MPI version 2.0, 2 processes
Count processors: 2 My rank is 0
time for set_epsilon = 0.365422 s
creating output file "./eps-000000.00.h5"...
on time step 2066 (time=103.3), 0.00193695 s/step
on time step 4152 (time=207.6), 0.00191805 s/step
on time step 6205 (time=310.25), 0.00194897 s/step
on time step 8307 (time=415.35), 0.00190318 s/step
Field time usage:
connnecting chunks: 0.024951 s
time stepping: 14.0125 s
communicating: 4.72859 s
outputting fields: 0.00907009 s
Fourier transforming: 0.0204064 s
everything else: 0.438375 s
_______________________________________________
meep-discuss mailing list
meep-discuss@ab-initio.mit.edu
http://ab-initio.mit.edu/cgi-bin/mailman/listinfo/meep-discuss
