Stwissel opened a new issue #268:
URL: https://github.com/apache/couchdb-nano/issues/268
<!--- Provide a general summary of the issue in the Title above -->
I need to process a large number of documents (>1M) made available in a
view. `viewAsStream` seems to be the logical choice, so I wrote:
```js
const source = nano.use(settings.source);
source
.viewAsStream("sortTimestamp", "date-device")
.on("error", (e) => console.error("error", e))
.pipe(aggregator);
```
Works like a charm stream all the data. I created a Writable stream:
```js
const { Writable } = require("stream");
const aggregator = new Writable({
construct(callback) {
callback();
},
write(chunk, encoding, callback) {
let buff = Buffer.from(chunk, encoding);
// Here I'm stuck
callback();
},
destroy() {
// No action
},
});
```
## Expected Behavior
<!--- If you're suggesting a change/improvement, tell us how it should work
-->
stream should emit consistent data, ideally one document at a time (as
buffer is fine)
## Current Behavior
<!--- If suggesting a change/improvement, explain the difference from
current behavior -->
The Buffer has "mostly" 5 lines of content (split `let lines =
buff.toString().split("\n")`) where `lines[0]` is either the beginning of the
result with count, skip and the beginning of the array -or- a single `,`.
Unfortunately not consistently. Somtimes a document is broken between chunks.
## Possible Solution
<!--- or ideas how to implement the addition or change -->
No clue - unfortunately
## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful
in the real world -->
Need to process a large number of documents >1M, so holding result in memory
isn't feasible.
Other processing suggestions are welcome
## Your Environment
<!--- Include as many relevant details about the environment you experienced
the bug in -->
* Version used:
- nano 9.0.3
- node 16.1.0
* Browser Name and version:
- command line app
* Operating System and version (desktop or mobile):
- macOS 10.15.7
*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
