Are you planning on putting *all* of the metadata in HTTP response headers (i.e. is it Content-Length type stuff)? The reason I ask is because you mentioned attaching the metadata in a similar manner to how request and filed work. If that's the case, you shouldn't need to tamper with the stream from your data source at all. Just put the headers in the response and pass the stream through.
The way I would approach this particular situation would be to use a promise library like Q <https://github.com/kriskowal/q> to establish promises for the stuff I need, one for the metadata and another for the tile data. These would both run asynchronously. Then, when both promises have been fulfilled I would invoke a response handler. I haven't tested the following, but it should give you a good idea of what I'm talking about. var http = require('http'); var q = require('q'); http.createServer(function (req, res) { var metadataPromise = getMetadata('file.json'); // async call var dataStreamPromise = getTileData('/path/to/tile.png'); // async call q.all([ metadataPromise, dataStreamPromise ]).spread(function (metadata, dataStream) { for (var headerName in metadata) res.headers[headerName] = metadata[headerName]; res.pipe(dataStream); }); }); Note in this example that the `dataStreamPromise` is a promise for a stream, not a buffer of the contents of that file. This may mean that you'll miss a few `data` events while you're waiting for the `metadataPromise` to resolve, so you should probably return a BufferedStream<https://github.com/mjijackson/bufferedstream>to make sure you get all the data. -- Michael Jackson @mjackson On Wed, Dec 12, 2012 at 1:27 AM, Paul Connolley <paul.connol...@gmail.com>wrote: > Hi there > > Long time lurker, first time poster. I've been working on a module for the > last couple of weeks as a bit of a training exercise. I´ve been digging in > to node for the last 6 months and I´m trying to make all my modules > streaming. The latest exercise is a reverse image proxy for mapping tiles > (Openlayers) with caching that actually would have some application in > frontend JS that I´ve written. > > I want to store ancillary data externally and then attach it to the stream > when piping (akin to how Request and Filed attach relevant header info) and > I obviously want to do this asynchronously. My initial thought was to > override the pipe method on the stream to load the metadata from file and > then call the real pipe once the ancillary data has been retrieved. My > concern is whether I would run the risk of losing the streamed image data > while waiting for the metadata to load. > > Would it be advisable to keep a buffer of incoming stream data while > waiting for the metadata and then emit from the buffer or would it be as > simple as pausing and resuming the incoming pipe? At the moment, I´m > storing JSON in flat files but I´ve just been refactoring so that I could > plug it in to redis or memcached or whatnot. > > As soon as I get to my desk, I will push my current code up to github in > case it´s necessary but any advice would definitely be welcomed. > Alternatively, if I´m taking the wrong route in solving this problem, any > criticism and redirection would be well received. > > Thanks, > connrs. > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
