Hello, Steve
On Friday, 13 October 2017 at 22:22:54 UTC, Steven Schveighoffer
wrote:
On 10/13/17 6:18 PM, ikod wrote:
On Friday, 13 October 2017 at 19:17:54 UTC, Steven
Schveighoffer wrote:
Eventually, something like this will be possible with
jsoniopipe (I need to update and release this too, it's
probably broken with some of the changes I just put into
iopipe). Hopefully combined with some sort of networking
library you could process a JSON stream without reading the
whole thing into memory.
This can be done with requests. You can ask not to load whole
content in memory, but instead produce input range, which will
continue to load data from server when you will be ready to
consume:
auto rq = Request();
rq.useStreaming = true;
auto rs = rq.get("http://httpbin.org/image/jpeg");
auto stream = rs.receiveAsRange();
while(!stream.empty) {
// stream.front contain next data portion
writefln("Received %d bytes, total received %d from
document legth %d", stream.front.length, rq.contentReceived,
rq.contentLength);
stream.popFront; // continue to load from server
}
Very nice, I will add a component to iopipe that converts a
"chunk-like" range like this into an iopipe source, as this is
going to be needed to interface with existing libraries. I
still will want to skip the middle man buffer at some point
though :)
Thanks!
-Steve
Just in order to have complete picture here - getContent returns
not just ubyte[], but more rich structure (which can be convered
to ubyte[] if needed). Basically it is an
immutable(immutable(ubyte)[]) and almost all data there are just
data received from network without any data copy.
There are more details and docs on
https://github.com/ikod/nbuff/blob/master/source/nbuff/buffer.d.
Main goal behind Buffer is to minimize data movement, but it also
support many range properties, as long as some internal optimized
methods.
Thanks,
Igor