I would like to ask for advice on scaling for the 96 atom amorphous system I am attempting to use hybrid calculations on. As it states below, my attempts with a (very) small system showed that the hybrid setup is functional and that there was a slowdown by a factor of about six between PBE and PBE0. In light of the comment I received that there there may be issues with size allocations for working with the 2x2x2 MP grid I was initially using, I attempted again to run a hybrid calculation with just one k-point at Gamma. In this case my machines file looked like:
lapw0:localhost:12 1:localhost:24 sagittarius-ib:24 draco-ib:24 libra-ib:24 granularity:1 extrafine:1 This calculation quickly finished the lapw0 calculation and has been running for several days for lapw1c. Each nodes has two 12 core Ivy Bridge CPUs on it and the nodes are interconnected via Infiband. I know that a 2x2x2 MP grid Vasp hybrid calculation took about 24 hours on the Archer supercomputer using 8 nodes which is approximately twice the number of CPUs that I have. The number of valence electrons in the pseudopotentials was significantly less though and did not include the d-states which I have included in my calculations. Would it be possible to hear a (wild is OK) guess of the amount of CPU hours required to complete a calculation on the 96 atom system? The MgO results are repeated below: As suggested I have calculated a hybrid functional for the MgO primitive cell. My .machines file was # lapw0:localhost:2 8:localhost:2 8:draco-ib:2 granularity extrafine:1 and my MgO.inhf file looks like 0.25 alpha T screened (T) or unscreened (F) 0.165 lambda 9 nband 6 gmax 3 lmaxe 3 lmaxv 1d-3 tolu For the regular PBE calculation (just run_lapw -p -in1new 2), a SCF loop took about 15 seconds. For the hybrid calculation with options “run_lapw -hf -p -in1new 2”, with the above case.inhf file (screening off), the calculation on the same structure took about 90 seconds/SCF loop, making the hybrid calculations about a factor of six slower. I encountered no errors with this run unlike my attempts with the 96 atom system. In response to the question asked about the size of aCGT.weighf, the size of the file was 117,020 bytes. Best wishes, Paul Fons On Mar 13, 2015, at 11:37 PM, t...@theochem.tuwien.ac.at<mailto:t...@theochem.tuwien.ac.at> wrote: What is the size of aCGT.weighhf? Is it empty? Also, before continuing further with your big system, it would be interesting to know if the same problem occurs with a very small system like MgO (on same machine and in MPI mode). Anyway, I still think that it is hopeless to apply hybrid functionals on such a big system. F. Tran On Fri, 13 Mar 2015, Paul Fons wrote: I attempted to run a SCF loop using a hybrid functional and have run into some problems. In my earlier try I had a incorrectly specified .machines file now I have addressed this problem. I also changed the SCRATCH environment variable to “./“ so that it points to the main directory for the calculation. I have run a PBE SCF loop to normal terminal for an amorphous cluster of 96 atoms of Cu-Ge-Te. I then ran the init_hf script and after setting the number of bands to 770 for my 1526 electron system, I set the MP grid to 2x2x2 for a total of four k-points. I then ran the command "run_lapw -hf -p -in1new 2”. The SCF loop ran through lapw0, lapw1, lapw2, core, and then crashed at the program hf. The MPI processes upon crashing reported the following error: forrtl: severe (67): input statement requires too much data, unit 26, file /usr/local/share/wien2k/Fons/aCGT/aCGT.weighhf I have no idea why the program failed this time. I am using the Intel compiler (15) and the Intel MPI environment (e.g. mpiexec.hydra) to launch parallel programs as can be seen in the “parallel_options” file. The only *.error files are those like “hf_1.error” to “hf_4.error” which contain the not particularly useful information “error in hf”. So the error occurred in the routine “hf”. I would be most grateful for any advice as to what to try next. I have included what I hope is relevant debugging information below. My parallel_options files (in all nodes) are mats...@gemini.a04.aist.go.jp<mailto:mats...@gemini.a04.aist.go.jp>:~/Wien2K>cat parallel_options setenv TASKSET "no" setenv USE_REMOTE 0 setenv MPI_REMOTE 0 setenv WIEN_GRANULARITY 1 setenv WIEN_MPIRUN "mpiexec.hydra -n _NP_ -machinefile _HOSTS_ _EXEC_" My .machines files is as follows: lapw0:localhost:12 1:localhost:12 1:localhost:12 1:draco-ib:12 1:draco-ib:12 granularity:1 extrafine:1 CONTENTS of :parallel ----------------------------------------------------------------- starting parallel lapw1 at Thu Mar 12 14:25:13 JST 2015 localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost(1) 25254.539u 519.601s 35:44.50 1201.8% 0+0k 8+882304io 0pf+0w localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost(1) 24889.112u 585.238s 35:41.95 1189.3% 0+0k 0+719488io 0pf+0w draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib(1) 0.034u 0.021s 32:40.68 0.0% 0+0k 0+0io 0pf+0w draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib(1) 0.035u 0.017s 32:39.14 0.0% 0+0k 0+0io 0pf+0w Summary of lapw1para: localhost k=0 user=0 wallclock=0 draco-ib k=0 user=0 wallclock=0 <- done at Thu Mar 12 15:01:00 JST 2015 ----------------------------------------------------------------- -> starting Fermi on gemini.a04.aist.go.jp<http://gemini.a04.aist.go.jp> at Thu Mar 12 15:28:19 JST 2015 -> starting parallel lapw2c at Thu Mar 12 15:28:20 JST 2015 localhost 389.940u 7.565s 0:36.04 1102.9% 0+0k 718416+253704io 0pf+0w localhost 347.944u 5.749s 0:31.67 1116.7% 0+0k 718528+199776io 0pf+0w draco-ib 0.029u 0.026s 0:33.86 0.1% 0+0k 8+0io 0pf+0w draco-ib 0.032u 0.020s 0:33.80 0.1% 0+0k 8+0io 0pf+0w Summary of lapw2para: localhost user=737.884 wallclock=67.71 draco-ib user=0.061 wallclock=67.66 <- done at Thu Mar 12 15:28:57 JST 2015 -> starting sumpara 4 on gemini.a04.aist.go.jp<http://gemini.a04.aist.go.jp> at Thu Mar 12 15:28:58 JST 2015 <- done at Thu Mar 12 15:29:27 JST 2015 ----------------------------------------------------------------- -> starting parallel hfc at Thu Mar 12 15:29:35 JST 2015 ** HF crashed at Thu Mar 12 15:29:40 JST 2015 ** check ERROR FILES! ———————————————————————————————— DAYFILE mats...@libra.a04.aist.go.jp<mailto:mats...@libra.a04.aist.go.jp>:/usr/local/share/wien2k/Fons/aCGT>cat aCGT.dayfile Calculating aCGT in /usr/local/share/wien2k/Fons/aCGT on gemini.a04.aist.go.jp<http://gemini.a04.aist.go.jp> with PID 46746 using WIEN2k_14.2 (Release 15/10/2014) in /home/matstud/Wien2K start (Thu Mar 12 11:22:09 JST 2015) with lapw0 (40/99 to go) cycle 1 (Thu Mar 12 11:22:09 JST 2015) (40/99 to go) lapw0 -grr -p (11:22:09) starting parallel lapw0 at Thu Mar 12 11:22:09 JST 2015 -------- .machine0 : 12 processors 745.365u 3.350s 1:05.09 1150.2% 0+0k 144+796936io 0pf+0w lapw0 -p (11:23:14) starting parallel lapw0 at Thu Mar 12 11:23:15 JST 2015 -------- .machine0 : 12 processors 620.682u 2.669s 0:54.15 1151.1% 0+0k 40+203264io 0pf+0w lapw1 -c (11:24:09) 20736.270u 146.444s 3:01:03.72 192.2% 0+0k 11992+5913840io 0pf+0w lapw1 -p -c (14:25:13) starting parallel lapw1 at Thu Mar 12 14:25:13 JST 2015 -> starting parallel LAPW1 jobs at Thu Mar 12 14:25:13 JST 2015 running LAPW1 in parallel mode (using .machines) 4 number_of_parallel_jobs localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost(1) 25254.539u 519.601s 35:44.50 1201.8% 0+0k 8+882304io 0pf+0w localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost localhost(1) 24889.112u 585.238s 35:41.95 1189.3% 0+0k 0+719488io 0pf+0w draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib(1) 0.034u 0.021s 32:40.68 0.0% 0+0k 0+0io 0pf+0w draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib draco-ib(1) 0.035u 0.017s 32:39.14 0.0% 0+0k 0+0io 0pf+0w Summary of lapw1para: localhost k=0 user=0 wallclock=0 draco-ib k=0 user=0 wallclock=0 50150.035u 1107.587s 35:46.80 2387.6% 0+0k 72+1603320io 0pf+0w lapw2 -c (15:01:00) 1728.441u 186.944s 27:18.21 116.9% 0+0k 5749640+254352io 0pf+0w lapw2 -p -c (15:28:18) running LAPW2 in parallel mode localhost 389.940u 7.565s 0:36.04 1102.9% 0+0k 718416+253704io 0pf+0w localhost 347.944u 5.749s 0:31.67 1116.7% 0+0k 718528+199776io 0pf+0w draco-ib 0.029u 0.026s 0:33.86 0.1% 0+0k 8+0io 0pf+0w draco-ib 0.032u 0.020s 0:33.80 0.1% 0+0k 8+0io 0pf+0w Summary of lapw2para: localhost user=737.884 wallclock=67.71 draco-ib user=0.061 wallclock=67.66 753.258u 15.562s 1:08.22 1126.9% 0+0k 2229568+654112io 0pf+0w lcore (15:29:27) 4.166u 0.370s 0:06.35 71.3% 0+0k 8+69416io 0pf+0w hf -p -c (15:29:34) running HF in parallel mode ** HF crashed! 0.987u 2.750s 0:06.26 59.5% 0+0k 1208+1344io 19pf+0w error: command /home/matstud/Wien2K/hfcpara -c hf.def failed stop error _______________________________________________ Wien mailing list Wien@zeus.theochem.tuwien.ac.at<mailto:Wien@zeus.theochem.tuwien.ac.at> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien SEARCH the MAILING-LIST at: http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html _______________________________________________ Wien mailing list Wien@zeus.theochem.tuwien.ac.at<mailto:Wien@zeus.theochem.tuwien.ac.at> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien SEARCH the MAILING-LIST at: http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
_______________________________________________ Wien mailing list Wien@zeus.theochem.tuwien.ac.at http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien SEARCH the MAILING-LIST at: http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
