help
2014-07-12 0:10 GMT+08:00 <[email protected]>: > Send Hdf-forum mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Hdf-forum digest..." > > > Today's Topics: > > 1. Chunking and Parallel HDF5 (Zmick, David) > 2. Re: Slow writing parallel HDF5 performance (for only one > variable) (Angel de Vicente) > 3. Re: Hdf-forum Digest, Vol 61, Issue 7 (Cy Li) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 10 Jul 2014 21:57:21 +0000 > From: "Zmick, David" <[email protected]> > To: "[email protected]" <[email protected]> > Subject: [Hdf-forum] Chunking and Parallel HDF5 > Message-ID: > <[email protected]> > Content-Type: text/plain; charset="us-ascii" > > Hello All, > > It seems to me that having a chunk cache is rather important in ensuring > performance with chunked storage. How do existing applications leverage > chunking and parallel HDF5 without the chunk cache? > > Also, the documentation seems to imply that there is some thought going > into raw dataset chunk caching with MPI. Is this true? > > Thanks! > David > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20140710/90fb8b9a/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Fri, 11 Jul 2014 00:08:49 +0100 > From: Angel de Vicente <[email protected]> > To: HDF Users Discussion List <[email protected]> > Subject: Re: [Hdf-forum] Slow writing parallel HDF5 performance (for > only one variable) > Message-ID: <[email protected]> > Content-Type: text/plain; charset=utf-8 > > Hi, > > Angel de Vicente <[email protected]> writes: > > I'm having a hard time trying to figure out what could be causing the > > slow I/O behaviour that I see: the same code (in Fortran) run in three > > different clusters behaves pretty similar in terms of I/O times, except > > for one of the variables in the code, where I get two orders of > > magnitude slower writes in one of machines (last timing data in the > > e-mail). So I hope that somebody with more in-depth knowledge of > > Parallel HDF5 can give me a hand with it. > > regarding this issue, I extracted only the relevant parts of the code > and I can reproduce the behaviour I was explaining in the previous > e-mail with a very simple code, which you can see at: > > http://pastebin.com/HjnS82Gp > > (to compile it I just do h5pfc -o phdf5write timing.f90 phdf5write.f90, > where timing.f90 is timing routine by Arjen Markus, available at > http://pastebin.com/480RmNET) > > The code generates a 4D array with the relevant sizes, and writes the > relevant part of the data to a file in three modes PMLX, PMLY, and > PMLZ (these replicate the ranks that I would get for X,Y and Z planes > respectively when creating a Cartesian topology with the relevant MPI > routines). > > The clusters where I run this code all have 16 cores per node. When I > run this test code in only 8 cores (nblocks set to 2), then the three > clusters behave similarly and there is no penalty for writing the > PMLZ. When I run it in 64 nodes, only PMLZ is heavily penalized, and > only very badly in one cluster, while badly in the other one. It looks > like there is some contention issue with the Parallel file system when > the cores span a number of nodes, but I certainly don't understand why > it only affects the PMLZ variable and not the PMLY, and why one of the > clusters doesn't seem to be affected. > > Is there something in the code something that is calling for trouble? > Any ideas/pointers/suggestions? > > Thanks a lot, > -- > ?ngel de Vicente > http://www.iac.es/galeria/angelv/ > > --------------------------------------------------------------------------------------------- > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protecci?n de > Datos, acceda a http://www.iac.es/disclaimer.php > WARNING: For more information on privacy and fulfilment of the Law > concerning the Protection of Data, consult > http://www.iac.es/disclaimer.php?lang=en > > > > > ------------------------------ > > Message: 3 > Date: Sat, 12 Jul 2014 00:10:26 +0800 > From: Cy Li <[email protected]> > To: [email protected] > Subject: Re: [Hdf-forum] Hdf-forum Digest, Vol 61, Issue 7 > Message-ID: > < > cabtswr9ziq1xus5cfk+uiq2ndwukgat9tewdsse4kcpdkmh...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > help > > > 2014-07-08 16:38 GMT+08:00 <[email protected]>: > > > Send Hdf-forum mailing list submissions to > > [email protected] > > > > To subscribe or unsubscribe via the World Wide Web, visit > > > > > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > > > > or, via email, send a message with subject or body 'help' to > > [email protected] > > > > You can reach the person managing the list at > > [email protected] > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of Hdf-forum digest..." > > > > > > Today's Topics: > > > > 1. Re: make check 2 (fereshteh komijani) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Tue, 8 Jul 2014 13:08:18 +0430 > > From: fereshteh komijani <[email protected]> > > To: HDF Users Discussion List <[email protected]> > > Subject: Re: [Hdf-forum] make check 2 > > Message-ID: > > <CAPy__Tg=DQe7xDgbZBTQ3jqs50Zpoqf7RQAchqUh7V8s8= > > [email protected]> > > Content-Type: text/plain; charset="utf-8" > > > > when i used > > [ @localost] gcc version > > gcc (GCC) 4.8.2 20131017 REDHAT (4.8.2_1) > > Now please help me > > cheers > > fereshte > > > > > > On Mon, Jul 7, 2014 at 9:43 PM, George N. White III <[email protected]> > > wrote: > > > > > Your report omitted the important detail of your compiler version. The > > > "dt_arith" failure looks > > > similar to <http://www.hdfgroup.org/HDF5/release/known_problems/> > > > "Conversion Tests Fail > > > on Ubuntu 11.10 with gcc 4.6.1", where you are advised to use > "CFLAGS=-O1 > > > ./configure". > > > > > > > > > On Mon, Jul 7, 2014 at 12:13 PM, fereshteh komijani < > > > [email protected]> wrote: > > > > > >> dear friend > > >> i cant solve my problem there fore i request to help me about solving > > >> following error in command *Make check* in following way: > > >> > > >> 1-cd /home/fkomijani/HDF5 1.8.9 > > >> > > >> 2- ./configure --with-zlib=/home/fkomijani/NETCD4.2.1.1 > > >> --prefix=//home/fkomijani/NETCD4.2.1.1 > > >> > > >> make check install > > >> > > >> but in this step i get the error massage like: > > >> > > >> Testing dt_arith > > >> ============================ > > >> dt_arith Test Log > > >> ============================ > > >> Testing non-aligned conversions (ALIGNMENT=1).... > > >> Testing query functions of compiler conversion > > >> PASSED > > >> Testing user-define and query functions of floating-point types > > >> PASSED > > >> Testing user-define and query functions of integer types > > >> PASSED > > >> Floating-point overflow cases will be tested. > > >> Testing noop float -> float conversions > > >> PASSED > > >> Testing noop double -> double conversions > > >> PASSED > > >> Testing noop long double -> long double conversions > > >> PASSED > > >> Testing hard normalized float -> double conversions > > >> PASSED > > >> Testing hard normalized double -> float conversions > > >> PASSED > > >> Testing hard normalized float -> long double conversions > > >> PASSED > > >> Testing hard normalized double -> long double conversions > > >> PASSED > > >> Testing hard normalized long double -> float conversions > > >> PASSED > > >> Testing hard normalized long double -> double conversions > > >> PASSED > > >> Testing hard denormalized float -> double conversions > > >> PASSED > > >> Testing hard denormalized double -> float conversions > > >> PASSED > > >> Testing hard denormalized float -> long double conversions > > >> PASSED > > >> Testing hard denormalized double -> long double conversions > > >> PASSED > > >> Testing hard denormalized long double -> float conversions > > >> PASSED > > >> Testing hard denormalized long double -> double conversions > > >> PASSED > > >> Testing hard special float -> double conversions > > >> PASSED > > >> Testing hard special double -> float conversions > > >> PASSED > > >> Testing hard special float -> long double conversions > > >> PASSED > > >> Testing hard special double -> long double conversions > > >> PASSED > > >> Testing hard special long double -> float conversions > > >> Child caught signal 11. > > >> Testing hard special long double -> double conversions > > >> Child caught signal 11. > > >> Testing hard signed char -> unsigned char conversions > > >> PASSED > > >> Testing hard signed char -> short conversions > > >> PASSED > > >> Testing hard signed char -> unsigned short conversions > > >> PASSED > > >> Testing hard signed char -> int conversions > > >> PASSED > > >> Testing hard signed char -> unsigned int conversions > > >> PASSED > > >> Testing hard signed char -> long long conversions > > >> PASSED > > >> Testing hard signed char -> unsigned long long conversions > > >> PASSED > > >> Testing hard unsigned char -> signed char conversions > > >> PASSED > > >> Testing hard unsigned char -> short conversions > > >> PASSED > > >> Testing hard unsigned char -> unsigned short conversions > > >> PASSED > > >> Testing hard unsigned char -> int conversions > > >> PASSED > > >> Testing hard unsigned char -> unsigned int conversions > > >> PASSED > > >> Testing hard unsigned char -> long long conversions > > >> PASSED > > >> Testing hard unsigned char -> unsigned long long conversions > > >> PASSED > > >> Testing hard short -> signed char conversions > > >> PASSED > > >> Testing hard short -> unsigned char conversions > > >> PASSED > > >> Testing hard short -> unsigned short conversions > > >> PASSED > > >> Testing hard short -> int conversions > > >> PASSED > > >> Testing hard short -> unsigned int conversions > > >> PASSED > > >> Testing hard short -> long long conversions > > >> PASSED > > >> Testing hard short -> unsigned long long conversions > > >> PASSED > > >> Testing hard unsigned short -> signed char conversions > > >> PASSED > > >> Testing hard unsigned short -> unsigned char conversions > > >> PASSED > > >> Testing hard unsigned short -> short conversions > > >> PASSED > > >> Testing hard unsigned short -> int conversions > > >> PASSED > > >> Testing hard unsigned short -> unsigned int conversions > > >> PASSED > > >> Testing hard unsigned short -> long long conversions > > >> PASSED > > >> Testing hard unsigned short -> unsigned long long conversions > > >> PASSED > > >> Testing hard int -> signed char conversions > > >> PASSED > > >> Testing hard int -> unsigned char conversions > > >> PASSED > > >> Testing hard int -> short conversions > > >> PASSED > > >> Testing hard int -> unsigned short conversions > > >> PASSED > > >> Testing hard int -> unsigned int conversions > > >> PASSED > > >> Testing hard int -> long long conversions > > >> PASSED > > >> Testing hard int -> unsigned long long conversions > > >> PASSED > > >> Testing hard unsigned int -> signed char conversions > > >> PASSED > > >> Testing hard unsigned int -> unsigned char conversions > > >> PASSED > > >> Testing hard unsigned int -> short conversions > > >> PASSED > > >> Testing hard unsigned int -> unsigned short conversions > > >> PASSED > > >> Testing hard unsigned int -> int conversions > > >> PASSED > > >> Testing hard unsigned int -> long long conversions > > >> PASSED > > >> Testing hard unsigned int -> unsigned long long conversions > > >> PASSED > > >> Testing hard long long -> signed char conversions > > >> PASSED > > >> Testing hard long long -> unsigned char conversions > > >> PASSED > > >> Testing hard long long -> short conversions > > >> PASSED > > >> Testing hard long long -> unsigned short conversions > > >> PASSED > > >> Testing hard long long -> int conversions > > >> PASSED > > >> Testing hard long long -> unsigned int conversions > > >> PASSED > > >> Testing hard long long -> unsigned long long conversions > > >> PASSED > > >> Testing hard unsigned long long -> signed char conversions > > >> PASSED > > >> Testing hard unsigned long long -> unsigned char conversions > > >> PASSED > > >> Testing hard unsigned long long -> short conversions > > >> PASSED > > >> Testing hard unsigned long long -> unsigned short conversions > > >> PASSED > > >> Testing hard unsigned long long -> int conversions > > >> PASSED > > >> Testing hard unsigned long long -> unsigned int conversions > > >> PASSED > > >> Testing hard unsigned long long -> long long conversions > > >> PASSED > > >> Testing hard signed char -> float conversions > > >> PASSED > > >> Testing hard signed char -> double conversions > > >> PASSED > > >> Testing hard unsigned char -> float conversions > > >> PASSED > > >> Testing hard unsigned char -> double conversions > > >> PASSED > > >> Testing hard short -> float conversions > > >> PASSED > > >> Testing hard short -> double conversions > > >> PASSED > > >> Testing hard unsigned short -> float conversions > > >> PASSED > > >> Testing hard unsigned short -> double conversions > > >> PASSED > > >> Testing hard int -> float conversions > > >> PASSED > > >> Testing hard int -> double conversions > > >> PASSED > > >> Testing hard unsigned int -> float conversions > > >> PASSED > > >> Testing hard unsigned int -> double conversions > > >> PASSED > > >> Testing hard long long -> float conversions > > >> PASSED > > >> Testing hard long long -> double conversions > > >> PASSED > > >> Testing hard unsigned long long -> float conversions > > >> PASSED > > >> Testing hard unsigned long long -> double conversions > > >> PASSED > > >> Testing hard signed char -> long double conversions > > >> PASSED > > >> Testing hard unsigned char -> long double conversions > > >> PASSED > > >> Testing hard short -> long double conversions > > >> PASSED > > >> Testing hard unsigned short -> long double conversions > > >> PASSED > > >> Testing hard int -> long double conversions > > >> PASSED > > >> Testing hard unsigned int -> long double conversions > > >> PASSED > > >> Testing hard long long -> long double conversions > > >> PASSED > > >> Testing hard unsigned long long -> long double conversions > > >> PASSED > > >> Testing hard normalized float -> signed char conversions > > >> PASSED > > >> Testing hard normalized double -> signed char conversions > > >> PASSED > > >> Testing hard normalized float -> unsigned char conversions > > >> PASSED > > >> Testing hard normalized double -> unsigned char conversions > > >> PASSED > > >> Testing hard normalized float -> short conversions > > >> PASSED > > >> Testing hard normalized double -> short conversions > > >> PASSED > > >> Testing hard normalized float -> unsigned short conversions > > >> PASSED > > >> Testing hard normalized double -> unsigned short conversions > > >> PASSED > > >> Testing hard normalized float -> int conversions > > >> PASSED > > >> Testing hard normalized double -> int conversions > > >> PASSED > > >> Testing hard normalized float -> unsigned int conversions > > >> PASSED > > >> Testing hard normalized double -> unsigned int conversions > > >> PASSED > > >> Testing hard normalized float -> long long conversions > > >> PASSED > > >> Testing hard normalized double -> long long conversions > > >> PASSED > > >> Testing hard normalized float -> unsigned long long conversions > > >> PASSED > > >> Testing hard normalized double -> unsigned long long conversions > > >> PASSED > > >> Testing hard normalized long double -> signed char conversions > > >> 0.07user 0.00system 0:00.16elapsed 47%CPU (0avgtext+0avgdata > > >> 4800maxresident)k > > >> 0inputs+264outputs (0major+14101minor)pagefaults 0swaps > > >> make[4]: *** [dt_arith.chkexe_] Error 1 > > >> make[4]: Leaving directory > > >> `/home/fkomijani/source_program/hdf5_1.8.9/test' > > >> make[3]: *** [build-check-s] Error 2 > > >> make[3]: Leaving directory > > >> `/home/fkomijani/source_program/hdf5_1.8.9/test' > > >> make[2]: *** [test] Error 2 > > >> make[2]: Leaving directory > > >> `/home/fkomijani/source_program/hdf5_1.8.9/test' > > >> make[1]: *** [check-am] Error 2 > > >> make[1]: Leaving directory > > >> `/home/fkomijani/source_program/hdf5_1.8.9/test' > > >> make: *** [check-recursive] Error 1 > > >> > > >> > > >> my system is fedora linux 20. > > >> many thanks > > >> fereshte > > >> > > >> -- > > >> ***Angel*** > > >> > > >> > > >> > > >> _______________________________________________ > > >> Hdf-forum is for HDF software users discussion. > > >> [email protected] > > >> > > >> > > > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > > >> Twitter: https://twitter.com/hdf5 > > >> > > > > > > > > > > > > -- > > > George N. White III <[email protected]> > > > Head of St. Margarets Bay, Nova Scotia > > > > > > _______________________________________________ > > > Hdf-forum is for HDF software users discussion. > > > [email protected] > > > > > > > > > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > > > Twitter: https://twitter.com/hdf5 > > > > > > > > > > > -- > > ***Angel*** > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > > > http://mail.lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20140708/bb091068/attachment.html > > > > > > > ------------------------------ > > > > Subject: Digest Footer > > > > _______________________________________________ > > Hdf-forum is for HDF software users discussion. > > [email protected] > > > > > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > > > > > > ------------------------------ > > > > End of Hdf-forum Digest, Vol 61, Issue 7 > > **************************************** > > > > > > -- > ??? > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20140712/db1835e2/attachment.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > > > ------------------------------ > > End of Hdf-forum Digest, Vol 61, Issue 12 > ***************************************** > -- 李重阳
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
