[Pw_forum] QE and mpich2, Linux
the -nimage and -npool were accepted as valid options when I put them at the end of the mpi command, as part of the espresso command. thanks for your help so far, I will keep trying to get this to work properly. Nick > -Original Message- > From: giannozz at democritos.it > Sent: Wed, 2 Feb 2011 08:48:46 +0100 > To: pw_forum at pwscf.org > Subject: Re: [Pw_forum] QE and mpich2, Linux > > > On Feb 2, 2011, at 3:54 , ac.rain at inbox.com wrote: > >> mpirun redirects to mpiexec which does not support >> options "-nimage" and "-npool" > > these options are interpreted by pw.x, not by mpirun or mpiexec > > P. > --- > Paolo Giannozzi, Dept of Chemistry&Physics&Environment, > Univ. Udine, via delle Scienze 208, 33100 Udine, Italy > Phone +39-0432-558216, fax +39-0432-558222 > > > > > ___ > Pw_forum mailing list > Pw_forum at pwscf.org > http://www.democritos.it/mailman/listinfo/pw_forum FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop! Check it out at http://www.inbox.com/earth
[Pw_forum] QE and mpich2, Linux
I reviewed that doc thank you, I wonder if it is a little out of date, mpirun redirects to mpiexec which does not support options "-nimage" and "-npool" , those are mentioned for Plane-wave tasks on page10. I am running it now in blocks of 4 cpus to match the number-combination requirement. 4 cores on the same system completes in 1h47m , 16 cores on the same system completes in 2h19m. Is this not going to get faster on a single system until we add more memory to the system? or perhaps to drop the number of cpus specified in mpiMachinefile.txt so it spreads a fewer number of jobs over a greater number of systems? or is the bottleneck to do with the local hard drives not being fast enough? I am not familiar with the maths equations/terminology mentioned in the document. thanks, nick > -Original Message- > From: giannozz at democritos.it > Sent: Mon, 31 Jan 2011 11:16:06 +0100 > To: pw_forum at pwscf.org > Subject: Re: [Pw_forum] QE and mpich2, Linux > > in order to take advantage of parallelization, some understanding of > how parallelization works in QE is needed. The user guide and this > document: > http://www.fisica.uniud.it/~giannozz/QE-Tutorial/tutorial_para.pdf > contain some info. Throwing in more processors will not by itself do > the job. > > P. > --- > Paolo Giannozzi, Dept of Chemistry&Physics&Environment, > Univ. Udine, via delle Scienze 208, 33100 Udine, Italy > Phone +39-0432-558216, fax +39-0432-558222 > > > > > ___ > Pw_forum mailing list > Pw_forum at pwscf.org > http://www.democritos.it/mailman/listinfo/pw_forum Send your photos by email in seconds... TRY FREE IM TOOLPACK at http://www.imtoolpack.com/default.aspx?rc=if3 Works in all emails, instant messengers, blogs, forums and social networks.
[Pw_forum] QE and mpich2, Linux
Dear QE team, I have been having trouble getting multi processing working on Linux systems. The tasks process fine, however it seems the more power I throw at it, the SLOWER it gets. For example 1hr45mins for 4 cores, 2hr10mins for 24 cores, and 2hr36mins for 3x systems with 24 cores (using the -n 62 option). mpiexec -n 62 -wdir /usr/local/espresso-4.2.1/tests -f /home/user/mpiMachinefile.txt ../bin/pw.x -inp /home/user/my_espresso_dir/A1N.newname.in Am I launching mpdboot with the right option? there are three systems. mpdboot -n 3 -f ~/mpiMachinefile.txt & I was first trying to get it working with PARA_PREFIX and PARA_POSTFIX but I had that hang issue with nothing actually happening so this method of running mpiexec from the terminal is getting very close. (mpiMachinefile.txt is shown below..) lserver3:24 lserver4:24 lserver5:24 any help appreciated, thank you, Nick - Linux Administrator. Share photos & screenshots in seconds... TRY FREE IM TOOLPACK at http://www.imtoolpack.com/default.aspx?rc=if1 Works in all emails, instant messengers, blogs, forums and social networks.
[Pw_forum] QE and mpich2, Linux
I am not using $TESTDIR, I am not using $name , I am not using < or >, I am not using -inp, I am not using pw.x directly but the script may be calling it. I have used some of these when trying to get it working but no luck yet so simplifying as much as possible to get something working at least. I do not understand your PARA_ lines in the below message, is it related to the two PARA lines in the script? or are you requesting that I modify the script elsewhere and approach this another way? How to run one of the test programs with mpich2 ? I don't mind which test it is.. but very frustrating staring at code and documentation all day without any progress just brick walls and errors... regards, Nick > -Original Message- > From: giannozz at democritos.it > Sent: Tue, 14 Dec 2010 11:30:45 +0100 > To: pw_forum at pwscf.org > Subject: Re: [Pw_forum] QE and mpich2, Linux > > Gabriele Sclauzero wrote: > >> This is not correct! Please retry with >> PARA_POSTFIX="-inp" > > no, this is not going to work. The correct way is to replace the "<" > in the two lines > $PARA_PREFIX $ESPRESSO_ROOT/bin/pw.x $PARA_POSTFIX < $TESTDIR/$name.in \ > > $TESTDIR/$name.out > with "-inp" > > P. > -- > Paolo Giannozzi, Democritos and University of Udine, Italy > ___ > Pw_forum mailing list > Pw_forum at pwscf.org > http://www.democritos.it/mailman/listinfo/pw_forum Send your photos by email in seconds... TRY FREE IM TOOLPACK at http://www.imtoolpack.com/default.aspx?rc=if3 Works in all emails, instant messengers, blogs, forums and social networks.
[Pw_forum] QE and mpich2, Linux
Hi Paolo thanks for your response. What do you mean exactly by "run interactively"? at first I tried running from the command line with mpi commands and then we moved to putting the info inside the PARA_PREFIX and PARA_PREFIX inside the actual script called "check-pw.x.j". to run this i have been doing "./check_pw.x.j" ,which is running interactively but only prints "Checking atom..." then sits there for long time, the machines working at 100% without doing anything. I tried to add "-inp file" to PARA_PREFIX I just get "unrecognized argument inp" then fail, when I add it to PARA_POSTFIX I don't get error but get the same results as previously with the hung output. (I have not required input file before? just trying to get a test working with the least amount of complexity possible). PARA_PREFIX="mpiexec -n 60 -f /home/user/mpiMachinefile.txt -wdir /usr/local/espresso-4.2.1/tests" PARA_POSTFIX="./check-pw.x.j -inp file" regards, Nick > -Original Message- > From: giannozz at democritos.it > Sent: Fri, 10 Dec 2010 08:20:45 +0100 > To: pw_forum at pwscf.org > Subject: Re: [Pw_forum] QE and mpich2, Linux > > Run interactively and look what happens. If nothing > happens, use option "-inp file" to read from file "file". > --- > Paolo Giannozzi, Dept of Chemistry&Physics, Univ. Udine > via delle Scienze 208, 33100 Udine, Italy > Phone +39-0432-558216, fax +39-0432-558222 > > > > > ___ > Pw_forum mailing list > Pw_forum at pwscf.org > http://www.democritos.it/mailman/listinfo/pw_forum FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop! Check it out at http://www.inbox.com/earth
[Pw_forum] QE and mpich2, Linux
Hi Paolo, Thank you for your assistance. that is looking better, This is the line I added to the script check-pw.x.j ... PARA_PREFIX="mpiexec -n 60 -f /home/user/mpiMachinefile.txt" when launched with "./check-pw.x.j" many 100% processes showed on each system. however I am not sure what it was doing because nothing else happened besides the text output "Checking atom...". Usually this test completes in less than an hour on 1 core, many cores should reduce the completion time, after a couple of hours I saw nothing more than "Checking atom..." so I pressed Ctrl+C on the terminal and it printed some more information... $ ./check-pw.x.j Checking atom...Killed by signal 2. [mpiexec at server4] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:99): one of the processes terminated badly; aborting [mpiexec at server4] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:18): bootstrap device returned error waiting for completion [mpiexec at server4] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:352): bootstrap server returned error waiting for completion [mpiexec at server4] main (./ui/mpich/mpiexec.c:294): process manager error waiting for completion FAILED with error condition! Input: atom.in, Output: atom.out, Reference: atom.ref Aborting $ I also tried adding "-wdir /usr/local/espresso-4.2.1/tests" at the end of PARA_PREFIX and gave the same result. I also tried uncommenting PARA_POSTFIX="./check-pw.x.j" and it gave the same result. thanks & regards, Nick > -Original Message- > From: giannozz at democritos.it > Sent: Wed, 8 Dec 2010 09:42:06 +0100 > To: pw_forum at pwscf.org > Subject: Re: [Pw_forum] QE and mpich2, Linux > > > On Dec 8, 2010, at 8:15 , ac.rain at inbox.com wrote: > >> $ mpiexec -f ~/mpiMachinefile.txt -n 10 -wdir /usr/local/ >> espresso-4.2.1/tests ./check-pw.x.j > > the correct way to start the tests in parallel is to edit PARA_PREFIX > and PARA_POSTFIX > in the "check-pw.x.j" script > > P. > --- > Paolo Giannozzi, Dept of Chemistry&Physics, Univ. Udine > via delle Scienze 208, 33100 Udine, Italy > Phone +39-0432-558216, fax +39-0432-558222 > > > > > ___ > Pw_forum mailing list > Pw_forum at pwscf.org > http://www.democritos.it/mailman/listinfo/pw_forum FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop! Check it out at http://www.inbox.com/earth
[Pw_forum] QE and mpich2, Linux
hi all, I am having troubles getting MPICH2 working with Quantum Espresso, it seems to be working somewhat but with errors. There may be an issue with my environment setup, some is guess-work as I couldn't find much documentation for espresso and mpich2. server4 is the system the user will submit from. server3 and server5 are extra nodes on the same subnet. mpich2 was built with gfortran support and ssh keys are setup with no message-Of-Day so nodes can talk properly. The espresso directory /usr/local/espresso-4.2.1 is located on server4 and NFS shared out to server3 and server5 with read/write mode (when I initially had NFS set to read-only, espresso was unable to write a temp file to the tests directory so I made it a read-write NFS export). This means all 3 systems have read-write access to the same espresso directory, is this correct? extra info: inside the examples/environment_variables file I have the line "TMP_DIR=/home/mpiexec_espresso_tmp" however no files were written there (each system has its own instance of that dir) it seems to just use /usr/local/espresso-4.2.1/tmp ,as I can see files from today's date. This is fine with me if it prefers to use the NFS partition. I tried messing with PARA_PREFIX and PARA_POSTFIX inside the variables file but I only ran into worse issues. here is my test command... mpiexec -f ~/mpiMachinefile.txt -n 10 -wdir /usr/local/espresso-4.2.1/tests ./check-pw.x.j here is the contents of my ~/mpiMachinefile.txt file... server3:24 server4:24 server5:24 I have also tried with -n 30 and it seemed to put most of the process on the first server in the list as expected, however I never saw 10 or more process using 100% of the core in 'top'. When there were many processes at once they were only using small percentage of resources, with one of the process using 100% of core/cpu. From my understanding there should be 24 different process per machine at 100%, depending on what it's doing. In summary: the errors are troubling, and I don't think the system(s) are using their full potential for simulating. below is the first part of command-line output, from the mpiexec using the -n 10 option. any advice appreciated, thanks, Nick - Linux Administrator $ mpiexec -f ~/mpiMachinefile.txt -n 10 -wdir /usr/local/espresso-4.2.1/tests ./check-pw.x.j Checking atom...Checking atom...Checking atom...Checking atom...Checking atom...Checking atom...Checking atom...Checking atom...Checking atom...Checking atom...passed Checking atom-lsda...passed Checking atom-pbe...discrepancy in pressure detected Reference: -14.44, You got: -14.43 Checking atom-sigmapbe...passed Checking atom-lsda...passed Checking atom-lsda...passed Checking atom-lsda...passed Checking atom-lsda...passed Checking atom-lsda...passed Checking atom-lsda...passed Checking atom-lsda...passed passed awk: cmd. line:6: fatal: cannot open file `atom.tmp' for reading (No such file or directory) /bin/rm: cannot remove `atom.tmp': No such file or directory Checking atom-lsda...Checking atom-lsda...STOP 2 FAILED with error condition! Input: atom-lsda.in, Output: atom-lsda.out, Reference: atom-lsda.ref Aborting STOP 2 STOP 2 FAILED with error condition! Input: atom-lsda.in, Output: atom-lsda.out, Reference: atom-lsda.ref Aborting FAILED with error condition! Input: atom-lsda.in, Output: atom-lsda.out, Reference: atom-lsda.ref Aborting STOP 2 FAILED with error condition! Input: atom-lsda.in, Output: atom-lsda.out, Reference: atom-lsda.ref Aborting STOP 2 FAILED with error condition! Input: atom-lsda.in, Output: atom-lsda.out, Reference: atom-lsda.ref Aborting STOP 2 FAILED with error condition! Input: atom-lsda.in, Output: atom-lsda.out, Reference: atom-lsda.ref Aborting discrepancy in number of scf iterations detected Reference: 7, You got: 11 Checking atom-pbe...discrepancy in number of scf iterations detected Reference: 7, You got: 11 Checking atom-pbe...discrepancy in pressure detected Reference: -14.44, You got: -14.43 Checking atom-sigmapbe...discrepancy in pressure detected Reference: -14.44, You got: -14.43 Checking atom-sigmapbe...STOP 2 FAILED with error condition! Input: atom-sigmapbe.in, Output: atom-sigmapbe.out, Reference: atom-sigmapbe.ref Aborting discrepancy in number of scf iterations detected Reference: 7, You got: 25 Checking atom-pbe...discrepancy in pressure detected Reference: -14.44, You got: -14.43 Checking atom-sigmapbe...discrepancy in total energy detected Reference: -31.491047, You got: 0.00 discrepancy in number of scf iterations detected Reference: 16, You got: discrepancy in pressure detected Reference: -15.02, You got: Checking berry...passed Checking berry, step 2 ...discrepancy in number of scf iterations detected Reference: 16, You got: 34 discrepancy in pressure detected Reference: -15.02, You got: -15.11 Checking berry...STOP 2 FAILED with error condition! Input: berry.in2, Output: berry.out2, Reference: berry.ref2 Aborting passed Checking berry, step
[Pw_forum] QE: install problem, 4.2.1
Hi Paolo, I put the values into the configure and all looks well, not a single error. much appreciated, thank you. ./configure MPIF90=mpif90 CC=gcc F90=gfortran F77=gfortran make all 2>&1 | tee makeall.log regards, nick - Linux Administrator > -Original Message- > From: giannozz at democritos.it > Sent: Wed, 10 Nov 2010 08:35:50 +0100 > To: pw_forum at pwscf.org > Subject: Re: [Pw_forum] QE: install problem, 4.2.1 > > > On Nov 10, 2010, at 4:31 , ac.rain at inbox.com wrote: > >> the following I put in there > > you shouldn't put there anything: configure will do the job > >> and it seemed to compile most things. >> >> MPIF90 = mpif90 >> F90 = gfortran >> CC = gcc >> F77 = gfortran >> >> the "make all" output looks good now except for some output lines >> that say "make[2]: g: Command not found". > > so it doesn't look good > >> So then I added a specification for "LD = ld" in the make.sys and >> it seemed to break it > > you shouldn't add anything to make.sys > > P. > --- > Paolo Giannozzi, Dept of Chemistry&Physics, Univ. Udine > via delle Scienze 208, 33100 Udine, Italy > Phone +39-0432-558216, fax +39-0432-558222 > > > > > ___ > Pw_forum mailing list > Pw_forum at pwscf.org > http://www.democritos.it/mailman/listinfo/pw_forum Send any screenshot to your friends in seconds... Works in all emails, instant messengers, blogs, forums and social networks. TRY IM TOOLPACK at http://www.imtoolpack.com/default.aspx?rc=if2 for FREE
[Pw_forum] QE: install problem, 4.2.1
Hi Paolo, yes that was the problem. it seems configure writes over make.sys, the following I put in there and it seemed to compile most things. MPIF90 = mpif90 F90 = gfortran CC = gcc F77 = gfortran the "make all" output looks good now except for some output lines that say "make[2]: g: Command not found". So then I added a specification for "LD = ld" in the make.sys and it seemed to break it, make all now shows many "undefined references to `'" messages before failing at libiotk again. do I need to have something in the LD_LIBS = section of the make.sys ? or should I put it back to how it was with a blank LD= and ignore the g: error. the LD_LIBS line is not specified, the LDFLAGS line is set to -g. thanks, nick > -Original Message- > From: giannozz at democritos.it > Sent: Tue, 09 Nov 2010 18:01:33 +0100 > To: pw_forum at pwscf.org > Subject: Re: [Pw_forum] QE: install problem, 4.2.1 > > ac.rain at inbox.com wrote: > >> export MPIF90=mpf90 > > I think this is the problem: "mpf90" is not in the path or not > existent and "configure" gets an empty string when it checks if > it exists and works > > P. > -- > Paolo Giannozzi, Democritos and University of Udine, Italy > ___ > Pw_forum mailing list > Pw_forum at pwscf.org > http://www.democritos.it/mailman/listinfo/pw_forum Send your photos by email in seconds... TRY FREE IM TOOLPACK at http://www.imtoolpack.com/default.aspx?rc=if3 Works in all emails, instant messengers, blogs, forums and social networks.
[Pw_forum] QE: install problem, 4.2.1
Hi all, I am having some troubles getting Quantum Espresso to build on a Linux system (CentOS 5.5) x86_64. my current issue is getting passed the iotk error... export CC=gcc export MPIF90=mpf90 export PATH=/usr/local/mpich2-install/bin:$PATH (to make sure mpf90 location is in the $PATH, mpich2 was compiled with gfortran support!) export F90=gfortran after ./configure ,the "make all" output is shown below... # make all test -d bin || mkdir bin if test -e extlibs/archive/iotk-1.1.beta.tar.gz ; then \ ( cd extlibs ; make libiotk) ; fi make[1]: Entering directory `/home/user/espresso-4.2.1/extlibs' if test ! -d ../S3DE; then \ (gzip -dc archive/iotk-1.1.beta.tar.gz | (cd ../; tar -xvf -)) ; \ if test -e install/Makefile_iotk; then \ (cp install/Makefile_iotk ../S3DE/iotk/src/Makefile); fi; \ if test -e install/iotk_config.h; then \ (cp install/iotk_config.h ../S3DE/iotk/include/iotk_config.h); fi; fi cd ../S3DE/iotk/src; make lib+util; make[2]: Entering directory `/home/user/espresso-4.2.1/S3DE/iotk/src' O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_base.f90 make[2]: O3: Command not found make[2]: [iotk_base.o] Error 127 (ignored) O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_attr_interf.f90 make[2]: O3: Command not found make[2]: [iotk_attr_interf.o] Error 127 (ignored) O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_error_interf.f90 make[2]: O3: Command not found make[2]: [iotk_error_interf.o] Error 127 (ignored) O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_misc_interf.f90 make[2]: O3: Command not found make[2]: [iotk_misc_interf.o] Error 127 (ignored) O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_str_interf.f90 make[2]: O3: Command not found make[2]: [iotk_str_interf.o] Error 127 (ignored) O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_attr+CHARACTER1_0.f90 make[2]: O3: Command not found make[2]: [iotk_attr+CHARACTER1_0.o] Error 127 (ignored) O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_fmt_interf.f90 make[2]: O3: Command not found make[2]: [iotk_fmt_interf.o] Error 127 (ignored) O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_xtox_interf.f90 make[2]: O3: Command not found make[2]: [iotk_xtox_interf.o] Error 127 (ignored) O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_attr+COMPLEX1_0.f90 make[2]: O3: Command not found make[2]: [iotk_attr+COMPLEX1_0.o] Error 127 (ignored) I omitted 2 pages of errors here that look very similar to the above and below lines O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_unit_list.f90 make[2]: O3: Command not found make[2]: [iotk_unit_list.o] Error 127 (ignored) O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_unit.f90 make[2]: O3: Command not found make[2]: [iotk_unit.o] Error 127 (ignored) O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_write.f90 make[2]: O3: Command not found make[2]: [iotk_write.o] Error 127 (ignored) O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__FFTW -I../include -c iotk_xtox.f90 make[2]: O3: Command not found make[2]: [iotk_xtox.o] Error 127 (ignored) ar ruv libiotk.a iotk_attr+CHARACTER1_0.o iotk_attr+COMPLEX1_0.o iotk_attr+COMPLEX1_3.o iotk_attr+COMPLEX1_6.o iotk_attr+COMPLEX2_0.o iotk_attr+COMPLEX2_3.o iotk_attr+COMPLEX2_6.o iotk_attr+COMPLEX3_0.o iotk_attr+COMPLEX3_3.o iotk_attr+COMPLEX3_6.o iotk_attr+COMPLEX4_0.o iotk_attr+COMPLEX4_3.o iotk_attr+COMPLEX4_6.o iotk_attr.o iotk_attr+INTEGER1_0.o iotk_attr+INTEGER1_3.o iotk_attr+INTEGER1_6.o iotk_attr+INTEGER2_0.o iotk_attr+INTEGER2_3.o iotk_attr+INTEGER2_6.o iotk_attr+INTEGER3_0.o iotk_attr+INTEGER3_3.o iotk_attr+INTEGER3_6.o iotk_attr+INTEGER4_0.o iotk_attr+INTEGER4_3.o iotk_attr+INTEGER4_6.o iotk_attr_interf.o iotk_attr+LOGICAL1_0.o iotk_attr+LOGICAL1_3.o iotk_attr+LOGICAL1_6.o iotk_attr+LOGICAL2_0.o iotk_attr+LOGICAL2_3.o iotk_attr+LOGICAL2_6.o iotk_attr+LOGICAL3_0.o iotk_attr+LOGICAL3_3.o iotk_attr+LOGICAL3_6.o iotk_attr+LOGICAL4_0.o iotk_attr+LOGICAL4_3.o iotk_attr+LOGICAL4_6.o iotk_attr+REAL1_0.o iotk_attr+REAL1_3.o iotk_attr+REAL1_6.o iotk_attr+REAL2_0.o iotk_attr+REAL2_3.o iotk_attr+REAL2_6.o iotk_attr+REAL3_0.o iotk_attr+REAL3_3.o iotk_attr+REAL3_6.o iotk_attr+REAL4_0.o iotk_attr+REAL4_3.o iotk_attr+REAL4_6.o iotk_base.o iotk_dat+CHARACTER1_0.o iotk_dat+CHARACTER1_3.o iotk_dat+CHARACTER1_6.o iotk_dat+COMPLEX1_0.o iotk_dat+COMPLEX1_3.o iotk_dat+COMPLEX1_6.o iotk_dat+COMPLEX2_0.o iotk_dat+COMPLEX2_3.o iotk_dat+COMPLEX2_6.o iotk_dat+COMPLEX3_0.o iotk_dat+COMPLEX3_3.o iotk_dat+COMPLEX3_6.o iotk_dat+COMPLEX4_0.o iotk_dat+COMPLEX4_3.o iotk_dat+COMPLEX4_6.o iotk_dat.o iotk_dat+INTEGER1_0.o iotk_dat+INTEGER1_3.o iotk
