2011/3/21, Henrik Löfås <henrik.lo...@fysik.uu.se>: > TBTrans is only parallized over k-points, meaning there is no reason trying > to run with more than one node when you only have one k-point.
In your context, the node's meaing is equal the cpu? Furthermore, according to your above explains, the NP's value for tbtrans computations should always equal to or less than the k-ponits of the system, am I right? Regards. > > Regards > Henrik > > 2011/3/21 <hongyi.z...@gmail.com> > >> > Mehmet Topsakal >> > Mon, 21 Mar 2011 03:41:26 -0700 >> > >> > This is a "too specific" problem and its solution is also "too >> > specific". >> > >> > You should find a solution by yourself..... >> > >> > >> > try this : >> > >> > -------------- >> > for i in `seq -w 0.0 0.1 1.8` >> > do >> > >> > cd $i >> > >> cat >> tbtrans.bsub <<EOF >> > APP_NAME=snode >> >> The snode in my case must use NP=16*N (N=1,2,3,...) cpus for a job. >> >> If I want to use arbitrary cpus for a job, I should use score queue. >> >> > NP=1 # you can increase this number and see what happens. >> > RUN="RAW" >> > >> > mpirun.lsf -np $NP /XXX/tbtrans < INPUT.fdf > OUTPUT_tbt.fdf >> > EOF >> >> The lsf administrator tell me that the mpirun.lsf in my case should be >> used >> like this: >> >> mpirun.lsf /XXX/tbtrans < INPUT.fdf > OUTPUT_tbt.fdf >> >> > >> > bsub < tbtrans.bsub >> >> I change the above line into this: >> >> chmod +x tbtrans.bsub >> bsub ./tbtrans.bsub >> >> > >> > cd .. >> > >> > done >> > -------------- >> >> The following is the lsf script based on your advice: >> >> -------------- >> #!/bin/bash >> for i in `seq -w 0.0 0.1 1.8` >> do >> >> cd $i >> >> cat > tbtrans.bsub <<EOF >> APP_NAME=score >> NP=1 # you can increase this number and see what happens. >> RUN="RAW" >> >> #mpirun.lsf -np $NP /XXX/tbtrans < INPUT.fdf > OUTPUT_tbt.fdf >> mpirun.lsf ~/software/siesta/tbtrans-3.0-rc2 < INPUT.fdf > OUTPUT_tbt.fdf >> EOF >> >> #bsub < tbtrans.bsub >> chmod +x ./tbtrans.bsub >> bsub ./tbtrans.bsub >> >> cd .. >> >> done >> -------------- >> >> Fortunately, I used this method successfully perform the job. But when I >> increase the NP's value, I meet the errors like this: >> >> ---------------------- >> WARNING !!!! >> Number of Nodes greater than number of k-Points !! >> Some nodes will be innactive in k-point parallelization !! >> >> ************************************************************************ >> K-Points On Nodes: >> >> Node Numb. K Pts. Tot. K Pts. >> 0 0 1 >> 1 1 1 >> ------------------------------------------------------------------------ >> K-Points Coordinates and Weights (Node,kx,ky,kz,wk),: >> >> 1 0.00000 0.00000 0.00000 1.00000 >> ************************************************************************ >> >> unit cell: >> 31.1805 0.0000 0.0000 >> 0.0000 15.1178 0.0000 >> 0.0000 0.0000 8.3203 >> unit cell: >> 31.1805 0.0000 0.0000 >> 0.0000 15.1178 0.0000 >> 0.0000 0.0000 8.3203 >> ============================================================== >> Projection Region: atoms : [ 33; 80] >> Projection Region: states: [ 1; 432] Tot: 432 >> >> Starting K-Point loop for spin: 1 >> >> Transport for spin -> 1 , kxy-point -> 1, of Node 1 >> kxy=(0.000 0.000 0.000 ),w= 1.0000 >> E T(E) TotDOS PDOS >> Signal 15 received. >> Signal 15 received. >> Job /home/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/mvapich_wrapper >> /home/users/nxuzhs/software/siesta/tbtrans-3.0-rc2 >> >> TID HOST_NAME COMMAND_LINE STATUS >> TERMINATION_TIME >> ===== ========== ================ ======================= >> =================== >> 00000 a324 /home/users/nxuz Exit (1) 03/21/2011 >> 22:22:48 >> 00001 a217 /home/users/nxuz Exit (174) 03/21/2011 >> 22:22:44 >> ----------------------- >> >> Attached please find the NP=1 and NP=2 's OUTPUT_tbt.fdf files for the >> bias >> voltage equal to 0.0. >> >> Any hints for this issue? >> >> Thanks in advance. >> >> Hongsheng. >> >> > >> > >> > >> > >> > On Sun, Mar 20, 2011 at 10:01 AM, <hongyi.z...@gmail.com> wrote: >> > >> >> Dear Siesta users, >> >> >> >> I've compiled the parallel version of (tran)siesta-3.0-rc2, then I goto >> the >> >> Util/TBTrans directory and type make to generate the tbtrans program. >> >> >> >> I want to know in this method, the tbtrans itself is parallel or >> >> serial? >> >> >> >> Now, I'm trying to run the example provided by M. Topsakal: >> >> >> >> http://unam.bilkent.edu.tr/mt2/transiesta/agnr8-transiesta >> >> >> >> I've successfully completed the bias voltages computations by using the >> >> following bias_script.sh script, which is based on the >> >> >> http://unam.bilkent.edu.tr/mt2/transiesta/agnr8-transiesta/input_only/step-2-IV/bias_script.sh >> >> : >> >> >> >> --------------- >> >> #!/bin/bash >> >> # >> >> #BSUB -a intelmpi # Bu satiri silmeyin >> >> #BSUB # LSF stdout dosyasi ismi .%J.out >> >> #BSUB # LSF stderr dosyasi ismi .%J.err >> >> #BSUB -q salim.ciraci.q # kuyruk ismi >> >> #BSUB -m karadeniz_e1 # -m unam_bladez OR -m >> >> n_anadolu >> >> #BSUB -n 8 # islemci sayisi 4 8 16 32 >> >> #BSUB -R "span[ptile=8]" # >> >> #BSUB -J a8-b2-yb # LSF is ismi >> >> #ncpu=8 >> >> >> >> >> ################################################################################ >> >> APP_NAME=snode >> >> NP=16 >> >> RUN="RAW" >> >> >> >> cp ../step-1-electrodes/elec.TSHS . # copy TSHS file from step-1 >> >> >> >> mkdir cont # read the comment at the end of this script. >> >> >> >> for i in `seq -w 0.0 0.1 1.8` >> >> do >> >> >> >> >> >> cp -r cont $i >> >> cd $i >> >> cp ../*.psf . >> >> cp ../elec.TSHS . >> >> cp ../POSITIONS.fdf . >> >> >> >> cat > INPUT.fdf <<EOF >> >> SystemName scat >> >> SystemLabel scat >> >> >> >> %block kgrid_Monkhorst_Pack >> >> 1 0 0 0.0 >> >> 0 1 0 0.0 >> >> 0 0 3 0.0 >> >> %endblock kgrid_Monkhorst_Pack >> >> >> >> >> >> xc.functional GGA >> >> xc.authors PBE >> >> >> >> MeshCutoff 300.00000000 Ry >> >> SolutionMethod Transiesta >> >> OccupationFunction MP >> >> ElectronicTemperature 300 K >> >> >> >> SpinPolarized F >> >> FixSpin F >> >> >> >> MaxSCFIterations 1000 >> >> #DM.Tolerance 1.d-5 >> >> DM.NumberPulay 6 >> >> DM.NumberBroyden 0 >> >> DM.MixingWeight 0.1000000000 >> >> DM.OccupancyTolerance 0.1000000000E-11 >> >> DM.NumberKick 0 >> >> DM.KickMixingWeight 0.5000000000 >> >> DM.Tolerance 0.1000000000E-03 >> >> >> >> UseSaveData T >> >> >> >> MD.NumCGsteps 0 >> >> MD.TypeOfRun CG >> >> MD.VariableCell F >> >> MD.MaxCGDispl 0.2000000000 Bohr >> >> MD.MaxForceTol 0.05 eV/Ang >> >> #MD.MaxStressTol 0.0001 eV/Ang**3 >> >> >> >> >> >> Diag.ParallelOverK F >> >> PAO.EnergyShift 50 meV >> >> PAO.SplitNorm 0.1500000000 >> >> PAO.BasisType split >> >> PAO.BasisSize SZP >> >> >> >> >> >> WriteMullikenPop 0 >> >> WriteBands F >> >> SaveRho F >> >> SaveElectrostaticPotential F >> >> SaveTotalPotential F >> >> WriteCoorXmol T >> >> >> >> >> >> >> >> %block ChemicalSpeciesLabel >> >> 1 6 C-GGA >> >> %endblock ChemicalSpeciesLabel >> >> >> >> %include POSITIONS.fdf >> >> >> >> >> >> # Transiesta information >> >> SolutionMethod Transiesta >> >> # GENGF OPTIONS >> >> TS.ComplexContour.Emin -28 eV >> >> TS.ComplexContour.NPoles 16 >> >> TS.ComplexContour.NCircle 16 >> >> TS.ComplexContour.NLine 10 >> >> # BIAS OPTIONS >> >> TS.biasContour.NumPoints 10 >> >> >> >> >> >> # TS OPTIONS >> >> TS.Voltage $i eV >> >> >> >> # TBT OPTIONS >> >> TS.TBT.Emin -3 eV >> >> TS.TBT.Emax +3 eV >> >> TS.TBT.NPoints 500 >> >> TS.TBT.NEigen 3 >> >> TS.TBT.Eta 0.000001 Ry >> >> >> >> # Write electrode hamiltonian >> >> TS.SaveHS .true. >> >> TS.SaveLead .true. >> >> >> >> >> >> >> >> # LEFT ELECTRODE >> >> TS.HSFileLeft ./elec.TSHS >> >> TS.NumUsedAtomsLeft 16 >> >> TS.BufferAtomsLeft 16 >> >> >> >> # RIGHT ELECTRODE >> >> TS.HSFileRight ./elec.TSHS >> >> TS.NumUsedAtomsRight 16 >> >> TS.BufferAtomsRight 16 >> >> >> >> EOF >> >> >> >> mpirun.lsf -np $NP ~/software/siesta/transiesta-3.0-rc2 < INPUT.fdf > >> >> OUTPUT.fdf >> >> #mpirun.lsf -np $ncpu /XXX/transiesta < INPUT.fdf > OUTPUT.fdf >> >> >> >> >> >> cd .. >> >> rm -rf cont >> >> mkdir cont >> >> >> >> cp ./$i/scat.TSDE ./$i/scat.TSHS ./$i/scat.DM cont # copy these >> >> files >> >> for continuation of the next bias step. >> >> >> >> >> >> done >> >> --------------- >> >> >> >> Now, I want to perform the jobs defined by tbtrans_script.sh ( >> >> >> http://unam.bilkent.edu.tr/mt2/transiesta/agnr8-transiesta/input_only/step-2-IV/tbtrans_script.sh >> ) >> >> as follows: >> >> >> >> -------------- >> >> for i in `seq -w 0.0 0.1 1.8` >> >> do >> >> >> >> cd $i >> >> >> >> /XXX/tbtrans < INPUT.fdf > OUTPUT_tbt.fdf # & # if you have more >> than >> >> 19 cpu-core, remove the first # on this line ! >> >> >> >> cd .. >> >> >> >> done >> >> -------------- >> >> >> >> In this step, I don't know how to submit the job to lsf system by using >> >> mpirun/mpirun.lsf just as I've done by using bias_script.sh. >> >> >> >> Furthermore, I want to know whether tbtrans support parallel runs or >> not? >> >> >> >> Thanks in advance. >> >> >> >> -- >> >> Hongsheng Zhao <hongyi.z...@gmail.com> >> >> School of Physics and Electrical Information Science, >> >> Ningxia University, Yinchuan 750021, China >> >> GnuPG DSA: 0xD108493 >> >> 2011-3-20 >> >> >> >> >> > >> > >> >> -- >> Hongsheng Zhao <hongyi.z...@gmail.com> >> School of Physics and Electrical Information Science, >> Ningxia University, Yinchuan 750021, China >> GnuPG DSA: 0xD108493 >> 2011-3-21 > -- Hongyi Zhao <hongyi.z...@gmail.com> Xinjiang Technical Institute of Physics and Chemistry Chinese Academy of Sciences GnuPG DSA: 0xD108493
