Hi, the database for the web application is a lot of measured data stream and data which come in on events. in a staging area the data will collected and then transferred into the web app data area. Some data must be collected via forms.
Thats the orchestra with a lot of hierarchical data which - you all know - not the best thing for traditional sql. Other people use the system for reporting, summary and statistics. We have a database version with high availability and a lot of work for stability and ... and ... We search for new ways ...???!! Thanks and regards Friedhelm -----Ursprüngliche Nachricht----- Von: Hdf-forum [mailto:[email protected]] Im Auftrag von [email protected] Gesendet: Montag, 7. August 2017 17:06 An: [email protected] Betreff: Hdf-forum Digest, Vol 98, Issue 6 Send Hdf-forum mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of Hdf-forum digest..." Today's Topics: 1. Re: hdf5 parallel h5py (Thomas Caswell) 2. Re: "File too large" error, seemingly related to MPI (Quincey Koziol) 3. Re: VCS URI (Quincey Koziol) 4. Re: hdf5 parallel h5py (John Readey) ---------------------------------------------------------------------- Message: 1 Date: Sat, 05 Aug 2017 16:48:23 +0000 From: Thomas Caswell <[email protected]> To: HDF Users Discussion List <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [Hdf-forum] hdf5 parallel h5py Message-ID: <caa48sf_lezc5trgmwgtwv03sboocvbcfeycp6pdrh_mvfp1...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" I would also look at h5serv (https://github.com/HDFGroup/h5serv) which puts a server in front of your hdf5 file. It serves as a single process owns it and serves as the serialization point which side-steps almost all of the multiple-client issues. h5pyd (https://github.com/HDFGroup/h5pyd) is a client of h5serv which has an identical high-level API to h5py. Tom On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <[email protected]> wrote: > It doesn?t sound like parallel HDF5 is what you are wanting to do here. > Parallel HDF5 is for an application where all applications are writing > in a very coordinated manner. All processes need to write the same > metadata to the file in ?collective? calls to the library, i.e. each > application makes the same calls using the same arguments in the same > order when making calls that modify the file metadata (creating files, > datasets or groups, writing attributes, etc. > <https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html>). > > It sounds like you have separate applications that are executing in a > somewhat independent manner. This will not work with parallel HDF5. > > > > Using the serial library, I can think of at least one approach that > might work well for you. HDF5 1.10 introduced a Single-writer > multi-reader (SWMR) mode = to open a file. Using a SWMR file for each > process, each process would open one file as the writer in SWMR mode, > and open the files from all the other processes as read-only in SWMR mode. > > > > http://docs.h5py.org/en/latest/swmr.html > > > > Jarom > > > > *From:* Hdf-forum [mailto:[email protected]] *On > Behalf Of *ISCaD GmbH > *Sent:* Thursday, August 3, 2017 1:19 AM > *To:* [email protected] > *Subject:* [Hdf-forum] hdf5 parallel h5py > > > > Dear all, > > > > I work on an web application which should store and receive the data > from an hdf5 file. > > > > Because several people work with this file and long running processes > I would like > > to use the mpi4py, h5py and HDF5. > > > > I work on debian linux stretch 64 Bit. > > > > What?s the way to parallel h5py. > > > > Thanks and regards > > > > Friedhelm Matten > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or > g > Twitter: https://twitter.com/hdf5 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachment s/20170805/4d84f060/attachment-0001.html> ------------------------------ Message: 2 Date: Mon, 7 Aug 2017 09:14:28 -0500 From: Quincey Koziol <[email protected]> To: HDF Users Discussion List <[email protected]> Subject: Re: [Hdf-forum] "File too large" error, seemingly related to MPI Message-ID: <[email protected]> Content-Type: text/plain; charset=utf-8 Hi Frederic, Could you give us some more details about your file and the call(s) you are making to HDF5? I can?t think of any reason that it would crash when creating a file like this, but something interesting could be going on? :-) Quincey > On Aug 7, 2017, at 5:28 AM, Frederic Perez <[email protected]> wrote: > > Hi, > > While writing significant amount of data in parallel, I obtain the > following error stack: > > HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 66: > #000: H5D.c line 194 in H5Dcreate2(): unable to create dataset > major: Dataset > minor: Unable to initialize object > #001: H5Dint.c line 453 in H5D__create_named(): unable to create and > link to dataset > major: Dataset > minor: Unable to initialize object > #002: H5L.c line 1638 in H5L_link_object(): unable to create new link > to object > major: Links > minor: Unable to initialize object > #003: H5L.c line 1882 in H5L_create_real(): can't insert link > major: Symbol table > minor: Unable to insert object > #004: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed > major: Symbol table > minor: Object not found > #005: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed > major: Symbol table > minor: Callback failed > #006: H5L.c line 1685 in H5L_link_cb(): unable to create object > major: Object header > minor: Unable to initialize object > #007: H5O.c line 3016 in H5O_obj_create(): unable to open object > major: Object header > minor: Can't open object > #008: H5Doh.c line 293 in H5O__dset_create(): unable to create dataset > major: Dataset > minor: Unable to initialize object > #009: H5Dint.c line 1060 in H5D__create(): can't update the metadata cache > major: Dataset > minor: Unable to initialize object > #010: H5Dint.c line 852 in H5D__update_oh_info(): unable to update > layout/pline/efl header message > major: Dataset > minor: Unable to initialize object > #011: H5Dlayout.c line 238 in H5D__layout_oh_create(): unable to > initialize storage > major: Dataset > minor: Unable to initialize object > #012: H5Dint.c line 1713 in H5D__alloc_storage(): unable to > initialize dataset with fill value > major: Dataset > minor: Unable to initialize object > #013: H5Dint.c line 1805 in H5D__init_storage(): unable to allocate > all chunks of dataset > major: Dataset > minor: Unable to initialize object > #014: H5Dchunk.c line 3575 in H5D__chunk_allocate(): unable to write > raw data to file > major: Low-level I/O > minor: Write failed > #015: H5Dchunk.c line 3745 in H5D__chunk_collective_fill(): unable to > write raw data to file > major: Low-level I/O > minor: Write failed > #016: H5Fio.c line 171 in H5F_block_write(): write through metadata > accumulator failed > major: Low-level I/O > minor: Write failed > #017: H5Faccum.c line 825 in H5F__accum_write(): file write failed > major: Low-level I/O > minor: Write failed > #018: H5FDint.c line 260 in H5FD_write(): driver write request failed > major: Virtual File Layer > minor: Write failed > #019: H5FDmpio.c line 1846 in H5FD_mpio_write(): MPI_File_write_at_all failed > major: Internal error (too specific to document in detail) > minor: Some MPI function failed > #020: H5FDmpio.c line 1846 in H5FD_mpio_write(): Other I/O error , > error stack: > ADIOI_NFS_WRITESTRIDED(672): Other I/O error File too large > major: Internal error (too specific to document in detail) > minor: MPI Error String > > > It basically claims that I am creating a file too large. But I > verified that the filesystem is capable of handling such a size. In my > case, the file is around 4 TB when it crashes. Where could this > problem come from? I thought HDF5 does not have a problem with very > large files. Plus, I am dividing the file in several datasets, and the > write operations work perfectly until, at some point, it crashes with > the errors above. > > Could it be an issue with HDF5? Or could it be an MPI limitation? I am > skeptic about the latter option: at the beginning, the program writes > several datasets inside the file succesfully (all the datasets being > the same size). If MPI was to blame, why wouldn't it crash at the > first write? > > Thank you for your help. > Fred > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or > g > Twitter: https://twitter.com/hdf5 ------------------------------ Message: 3 Date: Mon, 7 Aug 2017 09:15:41 -0500 From: Quincey Koziol <[email protected]> To: HDF Users Discussion List <[email protected]> Subject: Re: [Hdf-forum] VCS URI Message-ID: <[email protected]> Content-Type: text/plain; charset="us-ascii" Hi David, Sure, the git repo is here: https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse <https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse> Quincey > On Aug 7, 2017, at 5:27 AM, David Seifert <[email protected]> wrote: > > Hi HDF5 team and users, > is there any possibility for me to develop again the current VCS > sources for adding pkgconfig + Meson support? This makes development > and feature addition a lot easier, as there won't be any conflicts. > > Regards > David > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or > g > Twitter: https://twitter.com/hdf5 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachment s/20170807/bebdf029/attachment-0001.html> ------------------------------ Message: 4 Date: Mon, 7 Aug 2017 15:05:28 +0000 From: John Readey <[email protected]> To: HDF Users Discussion List <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [Hdf-forum] hdf5 parallel h5py Message-ID: <[email protected]> Content-Type: text/plain; charset="utf-8" Friedhelm, I?m not familiar with the specifics of your web app, but another possibility is to just have the app call h5serv directly. Anika @ NASA Goddard wrote a nice blog article on this approach: https://www.hdfgroup.org/2017/04/the-gfed-analysis-tool-an-hdf-server-implem entation/. John From: Hdf-forum <[email protected]> on behalf of Thomas Caswell <[email protected]> Reply-To: HDF Users Discussion List <[email protected]> Date: Saturday, August 5, 2017 at 9:48 AM To: HDF Users Discussion List <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [Hdf-forum] hdf5 parallel h5py I would also look at h5serv (https://github.com/HDFGroup/h5serv) which puts a server in front of your hdf5 file. It serves as a single process owns it and serves as the serialization point which side-steps almost all of the multiple-client issues. h5pyd (https://github.com/HDFGroup/h5pyd) is a client of h5serv which has an identical high-level API to h5py. Tom On Fri, Aug 4, 2017 at 12:01 PM Nelson, Jarom <[email protected]<mailto:[email protected]>> wrote: It doesn?t sound like parallel HDF5 is what you are wanting to do here. Parallel HDF5 is for an application where all applications are writing in a very coordinated manner. All processes need to write the same metadata to the file in ?collective? calls to the library, i.e. each application makes the same calls using the same arguments in the same order when making calls that modify the file metadata (creating files, datasets or groups, writing attributes, etc.<https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html>). It sounds like you have separate applications that are executing in a somewhat independent manner. This will not work with parallel HDF5. Using the serial library, I can think of at least one approach that might work well for you. HDF5 1.10 introduced a Single-writer multi-reader (SWMR) mode = to open a file. Using a SWMR file for each process, each process would open one file as the writer in SWMR mode, and open the files from all the other processes as read-only in SWMR mode. http://docs.h5py.org/en/latest/swmr.html Jarom From: Hdf-forum [mailto:[email protected]<mailto:hdf-forum-bounces@lists. hdfgroup.org>] On Behalf Of ISCaD GmbH Sent: Thursday, August 3, 2017 1:19 AM To: [email protected]<mailto:[email protected]> Subject: [Hdf-forum] hdf5 parallel h5py Dear all, I work on an web application which should store and receive the data from an hdf5 file. Because several people work with this file and long running processes I would like to use the mpi4py, h5py and HDF5. I work on debian linux stretch 64 Bit. What?s the way to parallel h5py. Thanks and regards Friedhelm Matten _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected]<mailto:[email protected]> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachment s/20170807/d8afdcac/attachment.html> ------------------------------ Subject: Digest Footer _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org ------------------------------ End of Hdf-forum Digest, Vol 98, Issue 6 **************************************** _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
