Re: [Wien] Speeding up calculations in parallel mose

[Pavel Ondračka]

Dear Viktor,

at 54 atoms, you should have enough k-points to run k-parallel which is
probably going to be the fastest option. So lapw1 + lapw2 k-parallel
and the rest OpenMP parallelized.

An example .machines file for your 8 cores could look like this:

1:localhost
1:localhost
1:localhost
1:localhost
1:localhost
1:localhost
1:localhost
1:localhost
omp_global:8
omp_lapw1:1
omp_lapw2:1

Additionally double check that you are indeed using the correct
libraries, go to you WIENROOT folder and do "ldd lapw1" you will see a
list of libraries it is inked against, make sure there is a
"libopenblas.so" and "libmvec.so" between them.
Also do the same for lapw0 and double check that it is compiled with
OpenMP (links to libgomp and links an OpenMP parallel openblas, I'm not
familiar with Ubuntu but it should be the library that comes with the
libopenblas-openmp package)).

That should hopefully do the trick. In general you can also test the
serial Wien2k benchmark. At single thread it should run around 20s on
your CPU and it should scale quite reasonably maybe up to 4-6 threads
with OMP_NUM_THREADS so that is also something you can check.

Best regards
Pavel

On Tue, 2023-08-22 at 12:30 +0300, Victor Zenou wrote:
> Dear Wien2k users!I’m investigating  a 54 tungsten atoms  supercell ,
> with 1 helium atom and 1 hydrogen atom (primitive cell) at different
> interstitial sites.  It takes ~ 46 hr per  calculation cycle, and
> half of it (~23 hr) in parallel mode.  The Wien2k version 23.2 was
> installed on Ubuntu 22.04.2 LTS. using gfortran and I set
> OMP_NUM_THREADS to  1, and used 2 parallel_jobs in the current work.
> The computer is build from  i7-10700 processor @ 2.90GHz (8 cores; 16
> Threads), 32 GB memory and 500 GB SSD. 
> In the past using the same computer , it took me ~ 14 hr per cycle
> for the same calculations, meaning 2-4 times faster than today. The
> wien2k version was 21.1, bur I can’t remember if the calculations
> were done in parallel, probably yes (I think  the number of parallel
> jobs was chosen automatically), and I think I set   OMP_NUM_THREADS
> to  4, but again I’m not sure.
> How can I speed up my calculations using the same computer?
> Best regards, Victor
> 
> ___
> Wien mailing list
> Wien@zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> SEARCH the MAILING-LIST at: 
> http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html

___
Wien mailing list
Wien@zeus.theochem.tuwien.ac.at
http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
SEARCH the MAILING-LIST at:  
http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html

Re: [Wien] Speeding up calculations in parallel mose

[Laurence Marks]

Beyond what Pavel said, you are still talking weeks for a calculation. A
critical issue is whether you are using sensible parameters. Many people
have used RKMAX 7 and 1000 kpts assuming that those are "right" -- they are
not. If your H and He have small RMTs then RKMAX should also be smaller.
Also, what is needed is a k-point density, so the number of k-points is
less for a supercell.

I suggest that you copy your case.struct file to a new directory then do
init_lapw in that directory. 23.2 will pick appropriate values for you, and
you can then copy them over.

Beyond that, get your boss to buy you a faster computer.

--
Professor Laurence Marks (Laurie)
Walter P Murphy Professor Emeritus
Department of Materials Science and Engineering, Northwestern University
www.numis.northwestern.edu
"Research is to see what everybody else has seen, and to think what nobody
else has thought" Albert Szent-Györgyi

On Tue, Aug 22, 2023, 04:32 Victor Zenou  wrote:

> Dear Wien2k users!
>
> I’m investigating  a 54 tungsten atoms  supercell ,  with 1 helium atom and 1 
> hydrogen atom (primitive cell) at different interstitial sites.  It takes ~ 
> 46 hr per  calculation cycle, and half of it (~23 hr) in parallel mode.  The 
> Wien2k version 23.2 was installed on Ubuntu 22.04.2 LTS. using gfortran and I 
> set OMP_NUM_THREADS to  1, and used 2 parallel_jobs in the current work. The 
> computer is build from  i7-10700 processor @ 2.90GHz (8 cores; 16 Threads), 
> 32 GB memory and 500 GB SSD. In the past using the same computer , it took me 
> ~ 14 hr per cycle for the same calculations, meaning 2-4 times faster than 
> today. The wien2k version was 21.1, bur I can’t remember if the calculations 
> were done in parallel, probably yes (I think  the number of parallel jobs was 
> chosen automatically), and I think I set   OMP_NUM_THREADS to  4, but again 
> I’m not sure.How can I speed up my calculations using the same computer?Best 
> regards, Victor
>
> ___
> Wien mailing list
> Wien@zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> SEARCH the MAILING-LIST at:
> http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
>
___
Wien mailing list
Wien@zeus.theochem.tuwien.ac.at
http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
SEARCH the MAILING-LIST at:  
http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html

Re: [Wien] Speeding up calculations in parallel mose

[Victor Zenou]

Dear Laurie and Pavel,
Thanks for your answer
1. In fact I'm using RKMAX 5.4 and 125 kpts
0.01 cc and 0.001 ec


2. About my boss, it took me more than one year to get a new computer. Please
define "fast"!
Victor


‫בתאריך יום ג׳, 22 באוג׳ 2023 ב-14:35 מאת ‪Laurence Marks‬‏ <‪
laurence.ma...@gmail.com‬‏>:‬

> Beyond what Pavel said, you are still talking weeks for a calculation. A
> critical issue is whether you are using sensible parameters. Many people
> have used RKMAX 7 and 1000 kpts assuming that those are "right" -- they are
> not. If your H and He have small RMTs then RKMAX should also be smaller.
> Also, what is needed is a k-point density, so the number of k-points is
> less for a supercell.
>
> I suggest that you copy your case.struct file to a new directory then do
> init_lapw in that directory. 23.2 will pick appropriate values for you, and
> you can then copy them over.
>
> Beyond that, get your boss to buy you a faster computer.
>
> --
> Professor Laurence Marks (Laurie)
> Walter P Murphy Professor Emeritus
> Department of Materials Science and Engineering, Northwestern University
> www.numis.northwestern.edu
> "Research is to see what everybody else has seen, and to think what nobody
> else has thought" Albert Szent-Györgyi
>
> On Tue, Aug 22, 2023, 04:32 Victor Zenou  wrote:
>
>> Dear Wien2k users!
>>
>> I’m investigating  a 54 tungsten atoms  supercell ,  with 1 helium atom and 
>> 1 hydrogen atom (primitive cell) at different interstitial sites.  It takes 
>> ~ 46 hr per  calculation cycle, and half of it (~23 hr) in parallel mode.  
>> The Wien2k version 23.2 was installed on Ubuntu 22.04.2 LTS. using gfortran 
>> and I set OMP_NUM_THREADS to  1, and used 2 parallel_jobs in the current 
>> work. The computer is build from  i7-10700 processor @ 2.90GHz (8 cores; 16 
>> Threads), 32 GB memory and 500 GB SSD. In the past using the same computer , 
>> it took me ~ 14 hr per cycle for the same calculations, meaning 2-4 times 
>> faster than today. The wien2k version was 21.1, bur I can’t remember if the 
>> calculations were done in parallel, probably yes (I think  the number of 
>> parallel jobs was chosen automatically), and I think I set   OMP_NUM_THREADS 
>> to  4, but again I’m not sure.How can I speed up my calculations using the 
>> same computer?Best regards, Victor
>>
>> ___
>> Wien mailing list
>> Wien@zeus.theochem.tuwien.ac.at
>> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
>> SEARCH the MAILING-LIST at:
>> http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
>>
> ___
> Wien mailing list
> Wien@zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> SEARCH the MAILING-LIST at:
> http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
>
___
Wien mailing list
Wien@zeus.theochem.tuwien.ac.at
http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
SEARCH the MAILING-LIST at:  
http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html

Re: [Wien] Speeding up calculations in parallel mose

[Gavin Abo]

Sometimes it can be better to have a cheap computer if you can remote 
connect to a high performance computing (hpc) [1] cluster.


There can be quite a bit of difference in the computing resources that a 
thousand dollar desktop computer provides compared at a million dollar 
hpc [2].


If your in academia, there are usually more options available for 
getting access to a hpc such as use your own university's or 
department's one if available, use a research center if there is one 
nearby, collaborate with a professor at another university that has one, 
get a research grant to pay to use a commercial one (e.g., Amazon Cloud 
[3]), and so on.  If in industry, it might be harder to get access to 
one unless you work at a large corporation that has one.


Desktop computers used to ship with 100 Mbps cards.   However, in new 
computers nowadays, you will likely find them to have 1 Gbps cards.  A 
Gigabit switch used to be expensive, but they have probably come down in 
price now.


If you can find some buddies (aka fellow researchers) at your local site 
and can connect them up to a Gigabit switch, while not as good as a hpc 
with InfiniBand [4], you may be able to Beowulf cluster [5] them to be 
better than a single desktop computer.


You might recall reading, since it has been in the archive and has been 
re-posted a few times, the post at [6] about how a Gigabit network of 
computers with i7 processors worked relatively well.


[1] https://en.wikipedia.org/wiki/High-performance_computing
[2] 
https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg22676.html

[3] https://aws.amazon.com/education/
[4] https://en.wikipedia.org/wiki/InfiniBand
[5] https://en.wikipedia.org/wiki/Beowulf_cluster
[6] 
https://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/msg09334.html


Kind Regards,

Gavin
WIEN2k user

On 8/22/2023 11:48 PM, Victor Zenou wrote:

Dear Laurie and Pavel,
Thanks for your answer
1. In fact I'm using RKMAX 5.4 and 125 kpts
0.01 cc and 0.001 ec


2. About my boss, it took me more than one year to get a new computer. 
Please define "fast"!

Victor


‫בתאריך יום ג׳, 22 באוג׳ 2023 ב-14:35 מאת ‪Laurence Marks‬‏ 
<‪laurence.ma...@gmail.com‬‏>:‬


Beyond what Pavel said, you are still talking weeks for a
calculation. A critical issue is whether you are using sensible
parameters. Many people have used RKMAX 7 and 1000 kpts assuming
that those are "right" -- they are not. If your H and He have
small RMTs then RKMAX should also be smaller. Also, what is needed
is a k-point density, so the number of k-points is less for a
supercell.

I suggest that you copy your case.struct file to a new directory
then do init_lapw in that directory. 23.2 will pick appropriate
values for you, and you can then copy them over.

Beyond that, get your boss to buy you a faster computer.

--
Professor Laurence Marks (Laurie)
Walter P Murphy Professor Emeritus
Department of Materials Science and Engineering, Northwestern
University
www.numis.northwestern.edu 
"Research is to see what everybody else has seen, and to think
what nobody else has thought" Albert Szent-Györgyi

On Tue, Aug 22, 2023, 04:32 Victor Zenou  wrote:

Dear Wien2k users!

I’m investigating a 54 tungsten atoms supercell , with 1
helium atom and 1 hydrogen atom (primitive cell) at different
interstitial sites. It takes ~ 46 hr per calculation cycle,
and half of it (~23 hr) in parallel mode. The Wien2k version
23.2 was installed on Ubuntu 22.04.2 LTS. using gfortran and I
set OMP_NUM_THREADS to 1, and used 2 parallel_jobs in the
current work. The computer is build from i7-10700 processor @
2.90GHz (8 cores; 16 Threads), 32 GB memory and 500 GB SSD.
In the past using the same computer , it took me ~ 14 hr per
cycle for the same calculations, meaning 2-4 times faster than
today. The wien2k version was 21.1, bur I can’t remember if
the calculations were done in parallel, probably yes (I think
the number of parallel jobs was chosen automatically), and I
think I set OMP_NUM_THREADS to 4, but again I’m not sure.
How can I speed up my calculations using the same computer?
Best regards, Victor

___
Wien mailing list
Wien@zeus.theochem.tuwien.ac.at
http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
SEARCH the MAILING-LIST at:
http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
___
Wien mailing list
Wien@zeus.theochem.tuwien.ac.at
http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
SEARCH the MAILING-LIST at:  
http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html

Re: [Wien] Speeding up calculations in parallel mose

[Peter Blaha]

Well, we get only part of the information each time.

You are using a 3x3x3 W supercell with H/He interstitials.

How did you initialize the calculation ?  Hopefully in the new batch 
type mode with a low precision (0 or 1) !!  If not, then:


RKMAX=5.4 seems still very big for the presumably small H/He spheres. 
What are the RMTs (set by the program).
If they are small (0.6-1.0 bohr), a much smaller RKMAX (3-4 ?) can be 
used. This may make your calculations 100 times faster.


125 k-points: are these the IBZ points (case.klist) - sounds quite too 
many-, or what you put as input to  x kgen ?

And as was said before: Of course you should use k-parallelization.

Remember: the benchmark test case has 36 atoms/cell and needs 20 
seconds/k-point. Your system is bigger (55 atoms), so make it 100 
seconds/k-point.

No Inversion symmetry ?, low packing, .. may cost another factor of 2-4

So this makes ~ 5 min/k-point, i.e. with  125 k-points and 8 cores one 
scf cycle should take roughly 1 hour.


Remember the general strategy:
First run with "minimum parameters" (init_lapw -prec 0), check timing 
and possible parallelization.
Save the calculation, increase k-points and/or RKmax and continue (never 
do a full init_lapw again, maybe use init_lapw -prec XX -nodstart)



Am 23.08.2023 um 07:48 schrieb Victor Zenou:

Dear Laurie and Pavel,
Thanks for your answer
1. In fact I'm using RKMAX 5.4 and 125 kpts
0.01 cc and 0.001 ec


2. About my boss, it took me more than one year to get a new computer. 
Please define "fast"!

Victor


‫בתאריך יום ג׳, 22 באוג׳ 2023 ב-14:35 מאת ‪Laurence Marks‬‏ 
<‪laurence.ma...@gmail.com ‬‏>:‬


Beyond what Pavel said, you are still talking weeks for a
calculation. A critical issue is whether you are using sensible
parameters. Maney people have used RKMAX 7 and 1000 kpts assuming
that those are "right" -- they are not. If your H and He have small
RMTs then RKMAX should also be smaller. Also, what is needed is a
k-point density, so the number of k-points is less for a supercell.

I suggest that you copy your case.struct file to a new directory
then do init_lapw in that directory. 23.2 will pick appropriate
values for you, and you can then copy them over.

Beyond that, get your boss to buy you a faster computer.

--
Professor Laurence Marks (Laurie)
Walter P Murphy Professor Emeritus
Department of Materials Science and Engineering, Northwestern University
www.numis.northwestern.edu 
"Research is to see what everybody else has seen, and to think what
nobody else has thought" Albert Szent-Györgyi

On Tue, Aug 22, 2023, 04:32 Victor Zenou mailto:za...@post.bgu.ac.il>> wrote:

Dear Wien2k users!

I’m investigating a 54 tungsten atoms supercell , with 1 helium
atom and 1 hydrogen atom (primitive cell) at different
interstitial sites. It takes ~ 46 hr per calculation cycle, and
half of it (~23 hr) in parallel mode. The Wien2k version 23.2
was installed on Ubuntu 22.04.2 LTS. using gfortran and I set
OMP_NUM_THREADS to 1, and used 2 parallel_jobs in the current
work. The computer is build from i7-10700 processor @ 2.90GHz (8
cores; 16 Threads), 32 GB memory and 500 GB SSD.
In the past using the same computer , it took me ~ 14 hr per
cycle for the same calculations, meaning 2-4 times faster than
today. The wien2k version was 21.1, bur I can’t remember if the
calculations were done in parallel, probably yes (I think the
number of parallel jobs was chosen automatically), and I think I
set OMP_NUM_THREADS to 4, but again I’m not sure.
How can I speed up my calculations using the same computer?
Best regards, Victor

___
Wien mailing list
Wien@zeus.theochem.tuwien.ac.at

http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien

SEARCH the MAILING-LIST at:
http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html 


___
Wien mailing list
Wien@zeus.theochem.tuwien.ac.at 
http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien

SEARCH the MAILING-LIST at:
http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html 



___
Wien mailing list
Wien@zeus.theochem.tuwien.ac.at
http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
SEARCH the MAILING-LIST at:  
http

Re: [Wien] Speeding up calculations in parallel mose

[Victor Zenou]

Hi

I didn't use the new batch type mode, as I'm sorry to say I've never heard
about it till now.

I used 0.85, 1.0 and 1.8 RMT's for H, He and W, respectively.

Yes, no inversion symmetry as I choose primitive cell, each time filled
with 1 H atom and/or a He atom, at different location/s.

I found that 125 k-points (63 IBZ) are good enough for the convergence
criterions I wanted: 0.001 ec and 0.01 cc.



I used:

W-H: Rmt*Kmax=3

W-He: Rmt*Kmax=3*1/0.85=3.5

W: Rmt*Kmax=3*1.8/0.85=6.35





However  Rmt*Kmax for W should be ~8, In that case:

W: Rmt*Kmax=8

W-H: Rmt*Kmax=8*0.85/1.8=3.77

W-He: Rmt*Kmax=8*1/1.8=4.44





I will check the strategy you suggested, and re-check my parameters.

Thanks,

Victor

‫בתאריך שבת, 26 באוג׳ 2023 ב-15:00 מאת ‪Peter Blaha‬‏ <‪
peter.bl...@tuwien.ac.at‬‏>:‬

> Well, we get only part of the information each time.
>
> You are using a 3x3x3 W supercell with H/He interstitials.
>
> How did you initialize the calculation ?  Hopefully in the new batch
> type mode with a low precision (0 or 1) !!  If not, then:
>
> RKMAX=5.4 seems still very big for the presumably small H/He spheres.
> What are the RMTs (set by the program).
> If they are small (0.6-1.0 bohr), a much smaller RKMAX (3-4 ?) can be
> used. This may make your calculations 100 times faster.
>
> 125 k-points: are these the IBZ points (case.klist) - sounds quite too
> many-, or what you put as input to  x kgen ?
> And as was said before: Of course you should use k-parallelization.
>
> Remember: the benchmark test case has 36 atoms/cell and needs 20
> seconds/k-point. Your system is bigger (55 atoms), so make it 100
> seconds/k-point.
> No Inversion symmetry ?, low packing, .. may cost another factor of 2-4
>
> So this makes ~ 5 min/k-point, i.e. with  125 k-points and 8 cores one
> scf cycle should take roughly 1 hour.
>
> Remember the general strategy:
> First run with "minimum parameters" (init_lapw -prec 0), check timing
> and possible parallelization.
> Save the calculation, increase k-points and/or RKmax and continue (never
> do a full init_lapw again, maybe use init_lapw -prec XX -nodstart)
>
>
> Am 23.08.2023 um 07:48 schrieb Victor Zenou:
> > Dear Laurie and Pavel,
> > Thanks for your answer
> > 1. In fact I'm using RKMAX 5.4 and 125 kpts
> > 0.01 cc and 0.001 ec
> >
> >
> > 2. About my boss, it took me more than one year to get a new computer.
> > Please define "fast"!
> > Victor
> >
> >
> > ‫בתאריך יום ג׳, 22 באוג׳ 2023 ב-14:35 מאת ‪Laurence Marks‬‏
> > <‪laurence.ma...@gmail.com ‬‏>:‬
> >
> > Beyond what Pavel said, you are still talking weeks for a
> > calculation. A critical issue is whether you are using sensible
> > parameters. Maney people have used RKMAX 7 and 1000 kpts assuming
> > that those are "right" -- they are not. If your H and He have small
> > RMTs then RKMAX should also be smaller. Also, what is needed is a
> > k-point density, so the number of k-points is less for a supercell.
> >
> > I suggest that you copy your case.struct file to a new directory
> > then do init_lapw in that directory. 23.2 will pick appropriate
> > values for you, and you can then copy them over.
> >
> > Beyond that, get your boss to buy you a faster computer.
> >
> > --
> > Professor Laurence Marks (Laurie)
> > Walter P Murphy Professor Emeritus
> > Department of Materials Science and Engineering, Northwestern
> University
> > www.numis.northwestern.edu 
> > "Research is to see what everybody else has seen, and to think what
> > nobody else has thought" Albert Szent-Györgyi
> >
> > On Tue, Aug 22, 2023, 04:32 Victor Zenou  > > wrote:
> >
> > Dear Wien2k users!
> >
> > I’m investigating a 54 tungsten atoms supercell , with 1 helium
> > atom and 1 hydrogen atom (primitive cell) at different
> > interstitial sites. It takes ~ 46 hr per calculation cycle, and
> > half of it (~23 hr) in parallel mode. The Wien2k version 23.2
> > was installed on Ubuntu 22.04.2 LTS. using gfortran and I set
> > OMP_NUM_THREADS to 1, and used 2 parallel_jobs in the current
> > work. The computer is build from i7-10700 processor @ 2.90GHz (8
> > cores; 16 Threads), 32 GB memory and 500 GB SSD.
> > In the past using the same computer , it took me ~ 14 hr per
> > cycle for the same calculations, meaning 2-4 times faster than
> > today. The wien2k version was 21.1, bur I can’t remember if the
> > calculations were done in parallel, probably yes (I think the
> > number of parallel jobs was chosen automatically), and I think I
> > set OMP_NUM_THREADS to 4, but again I’m not sure.
> > How can I speed up my calculations using the same computer?
> > Best regards, Victor
> >
> > ___
> > Wie

Re: [Wien] Speeding up calculations in parallel mose

[Victor Zenou]

I started a new calculation with "minimum parameters" using init_lapw -prec
0, which suggested the following parameters:

RMT*Kmax=5

64 k-points (6 IBZ)



 I did parallel calcualtion using:
1:localhost
1:localhost
1:localhost
1:localhost
1:localhost
1:localhost
1:localhost
1:localhost
granularity:1
extrafine:1
omp_global:8
omp_lapw1:1omp_lapw2:1

In few hours I got 6 full SCF cycles with average of ~38 minutes per cycle

Victor
___
Wien mailing list
Wien@zeus.theochem.tuwien.ac.at
http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
SEARCH the MAILING-LIST at:  
http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html