On Wed, Dec 22, 2010 at 5:40 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > On Dec 22, 2010, at 9:55 AM, Yongjun Chen wrote: > > > > > Satish, > > > > I have reconfigured the PETSC with ?download-mpich=1 and > ?with-device=ch3:sock. The results show that the speed up can now remain > increasing when computing cores increase from 1 to 16. However, the maximum > speed up is still only around 6.0 with 16 cores. The new log files can be > found in the attachment. > > > > > > (1) > > > > I checked the configuration of the first server again. This server is a > shared-memory computer, with > > > > Processors: 4 CPUS * 4Cores/CPU, with each core 2500MHz > > > > Memories: 16 *2 GB DDR2 333 MHz, dual channel, data width 64 bit, so the > memory Bandwidth for 2 memories is 64/8*166*2*2=5.4GB/s. > > Wait a minute. You have 16 cores that share 5.4 GB/s???? This is not > enough for iterative solvers, in fact this is absolutely terrible for > iterative solvers. You really want 5.4 GB/s PER core! This machine is > absolutely inappropriate for iterative solvers. No package can give you good > speedups on this machine. > > Barry > Barry, there are 16 memories, every 2 memories make up one dual channel, thus in this machine there are 8 dual channel, each dual channel has the memory bandwidth 5.4GB/s. Yongjun > > > > > It seems that each core can get 2.7GB/s memory bandwidth which can > fulfill the basic requirement for sparse iterative solvers. > > > > Is this correct? Does the shared-memory type of computer have no benefit > for PETSC when the memory bandwidth is limited? > > > > > > (2) > > > > Beside, we would like to continue our work by employing a matrix > partitioning / reordering algorithm, such as Metis or ParMetis, to improve > the speed up performance of the program. (The current program works without > any matrix decomposition.) > > > > > > Matt, as you said in > http://lists.mcs.anl.gov/pipermail/petsc-users/2007-January/001017.html,?Reordering > a matrix can result in fewer iterations for an iterative > solver?. > > > > Do you think the matrix partitioning/reordering will work for this > program? Or any further suggestions? > > > > > > Any comments are very welcome! Thank you! > > > > > > > > > > > > > > > > > > On Mon, Dec 20, 2010 at 11:04 PM, Satish Balay <balay at mcs.anl.gov> > wrote: > > On Mon, 20 Dec 2010, Yongjun Chen wrote: > > > > > Matt, Barry, thanks a lot for your reply! I will try mpich hydra > firstly and > > > see what I can get. > > > > hydra is just the process manager. > > > > Also --download-mpich uses a slightly older version - with > > device=ch3:sock for portability and valgrind reasons [development] > > > > You might want to install latest mpich manually with the defaut > > device=ch3:nemsis and recheck.. > > > > satish > > > > > > > > > <log_ch3sock_jacobi_bicg_4cpus.txt><log_ch3sock_jacobi_bicg_8cpus.txt><log_ch3sock_jacobi_bicg_12cpus.txt><log_ch3sock_jacobi_bicg_16cpus.txt> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20101222/e62fe60b/attachment.htm>
