Hey Ben, Thanks for the link. WebSockets are also worth exploring as well for large data transfers (looks like most recent browsers are supporting WebSockets).
Another tack is to ask "what does the page intend to do with the data anyway?". If say it is going to be displayed in a grid, it may make sense to dynamically fetch the data as users scroll around in the data rather than grabbing the entire dataset at once. The amount of values displayed in any one view of a datagrid is very small. John On Tuesday, December 2, 2014 4:13:46 AM UTC-8, Ben Jeffery wrote: > > Hi John, > > On the issue of encodings I've recently been using arraybuffers to > transfer HDF5 array subsets to the browser. ( > https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Sending_and_Receiving_Binary_Data) > > If the dtype is one supported by JS typed arrays, the array doesn't need to > be parsed. > Thought it was worth mentioning as for use cases with large arrays on fast > connections this method can be a good option. > > Thanks, > Ben > > On Tuesday, November 11, 2014 9:27:37 PM UTC, John Readey wrote: >> >> Hey Ray, >> >> For this first release, the focus will be mostly on the API >> definition rather than performance. For example, data is being sent as >> json formatted text. I don’t think it should be an issue to support BASE64 >> encoding for data read/writes in a future release. The client can specify >> the desired format in the Content-type http header. >> >> Similarly, I’m not doing anything special for reader/writer >> concurrency — the server is serializing all the requests. Clearly not >> suitable for a production service that will be seeing a lot of traffic. >> >> I’d be interested in hearing what performance requirements people have >> for an HDF server: bandwidth in/out, latency, request volume, etc. >> Depending on the specifics, there are different approaches for achieving >> performance targets. >> >> I hadn’t heard about the issue with ever-growing hdf5 files. Well, >> one nice aspect of the server-based approach is that you can consolidate >> any maintenance workflows. E.g. Periodically running h5repack on files in >> the server. >> >> John >> >> From: Ray Polachikov <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: Tuesday, November 11, 2014 at 4:47 AM >> To: "[email protected]" <[email protected]> >> Cc: "[email protected]" <[email protected]> >> Subject: Re: ANN: HDF5 for Python 2.4.0 BETA >> >> Hi John and Stuart, >> >> Thanks for the hint. I'm aware of this limitation. The wrapper classes do >> open/close the underlying file for every single operation. I found the >> overhead of this to be negligible (relative to the actual I/O operations). >> >> HDF5 Server sounds promising. It's great that some progress is being made >> in this area. I experimented with Array-based database servers such as >> SciDB, but – to date – data I/O is so much slower than with hdf5. One >> problem being that the SciDB Python-API is HTTP-based and, hence, numerical >> data is encoded as text. >> Very much looking forward to seeing your code. I wonder how you dealt >> with those reader/writer concurrency issues. I also wonder if you found a >> solution to the problem that deleting nodes in an hdf5 file does not affect >> file size, i.e., files are ever-growing. In my opinion, this is a nasty >> limitation of hdf5. >> >> Ray >> >> -- >> You received this message because you are subscribed to the Google Groups >> "h5py" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >>
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
