[ https://issues.apache.org/jira/browse/AVRO-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960663#comment-16960663 ]
ASF subversion and git services commented on AVRO-2562: ------------------------------------------------------- Commit a684f3e0b1859ba05c17b18407c2e9905701079a in avro's branch refs/heads/master from Kengo Seki [ https://gitbox.apache.org/repos/asf?p=avro.git;h=a684f3e ] AVRO-2562: Expose the JavaScript BlockEncoder's piped stream (#650) > Expose the JavaScript BlockEncoder's piped stream > ------------------------------------------------- > > Key: AVRO-2562 > URL: https://issues.apache.org/jira/browse/AVRO-2562 > Project: Apache Avro > Issue Type: New Feature > Components: javascript > Reporter: Kengo Seki > Assignee: Kengo Seki > Priority: Major > > The "createFileEncoder" test for the JavaScript implementation is flaky. It > sometimes fails in docker mode, as follows: > {code} > ~/repos/avro$ git checkout master > ~/repos/avro$ ./build.sh docker > sekikn@13d38772b8e6:~/avro$ ./build.sh veryclean > sekikn@13d38772b8e6:~/avro$ cd lang/js > sekikn@13d38772b8e6:~/avro/lang/js$ node -v > v6.17.1 > sekikn@13d38772b8e6:~/avro/lang/js$ npm i > sekikn@13d38772b8e6:~/avro/lang/js$ for i in $(seq 1 100); do echo "#${i}"; > if ! npm t; then break; fi; done > #1 > > avro-js@1.9.0 test /home/sekikn/avro/lang/js > > mocha > > ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ > > ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ > > ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ > 381 passing (557ms) > (snip) > #7 > > avro-js@1.9.0 test /home/sekikn/avro/lang/js > > mocha > > ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ > > ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ > > ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ > 380 passing (554ms) > 1 failing > 1) files createFileEncoder: > Uncaught AssertionError: 0 == 2 > + expected - actual > -0 > +2 > > at BlockDecoder.<anonymous> (test/test_files.js:574:18) > at endReadableNT (_stream_readable.js:978:12) > at _combinedTickCallback (internal/process/next_tick.js:80:11) > at process._tickCallback (internal/process/next_tick.js:104:9) > npm ERR! Test failed. See above for more details. > {code} > And it always fails with Node.js v8.10.0, which is installed by apt-get on > Ubuntu 18.04 LTS. > {code} > ~/repos/avro$ git checkout master > ~/repos/avro$ cd lang/js > ~/repos/avro/lang/js$ ./build.sh test > > avro-js@1.9.0 cover /home/sekikn/repos/avro/lang/js > > istanbul cover _mocha > > ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ > > ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ > > ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ > 380 passing (537ms) > 1 failing > 1) files createFileEncoder: > Uncaught AssertionError [ERR_ASSERTION]: 0 == 2 > + expected - actual > -0 > +2 > > at BlockDecoder.<anonymous> (test/test_files.js:574:18) > at endReadableNT (_stream_readable.js:1064:12) > at _combinedTickCallback (internal/process/next_tick.js:138:11) > at process._tickCallback (internal/process/next_tick.js:180:9) > ============================================================================= > Writing coverage object > [/home/sekikn/repos/avro/lang/js/coverage/coverage.json] > Writing coverage reports at [/home/sekikn/repos/avro/lang/js/coverage] > ============================================================================= > =============================== Coverage summary > =============================== > Statements : 100% ( 2529/2529 ) > Branches : 100% ( 996/996 ) > Functions : 100% ( 414/414 ) > Lines : 100% ( 2443/2443 ) > ================================================================================ > npm ERR! Linux 5.0.0-25-generic > npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "cover" > npm ERR! node v8.10.0 > npm ERR! npm v3.5.2 > npm ERR! code ELIFECYCLE > npm ERR! avro-js@1.9.0 cover: `istanbul cover _mocha` > npm ERR! Exit status 1 > npm ERR! > npm ERR! Failed at the avro-js@1.9.0 cover script 'istanbul cover _mocha'. > npm ERR! Make sure you have the latest version of node.js and npm installed. > npm ERR! If you do, this is most likely a problem with the avro-js package, > npm ERR! not with npm itself. > npm ERR! Tell the author that this fails on your system: > npm ERR! istanbul cover _mocha > npm ERR! You can get information on how to open an issue for this project > with: > npm ERR! npm bugs avro-js > npm ERR! Or if that isn't available, you can get their info via: > npm ERR! npm owner ls avro-js > npm ERR! There is likely additional logging output above. > npm ERR! Please include the following file with any support request: > npm ERR! /home/sekikn/repos/avro/lang/js/npm-debug.log > {code} > In this test, {{createFileDecoder()}} is called in the event listener for the > {{'finish'}} event emitted from the {{encoder}} object. > {code:title=test/test_files.js} > 553 test('createFileEncoder', function (cb) { > 554 var type = createType({ > 555 type: 'record', > 556 name: 'Person', > 557 fields: [ > 558 {name: 'name', type: 'string'}, > 559 {name: 'age', type: 'int'} > 560 ] > 561 }); > 562 var path = tmp.fileSync().name; > 563 var encoder = files.createFileEncoder(path, type); > 564 encoder.write({name: 'Ann', age: 32}); > 565 encoder.end({name: 'Bob', age: 33}); > 566 var n = 0; > 567 encoder.on('finish', function () { > 568 files.createFileDecoder(path) > 569 .on('data', function (obj) { > 570 n++; > 571 assert(type.isValid(obj)); > 572 }) > 573 .on('end', function () { > 574 assert.equal(n, 2); > 575 cb(); > 576 }); > 577 }); > 578 }); > {code} > And the {{encoder}} object has the underlying writable stream created by > {{createWriteStream()}}. > {code:title=lib/files.js} > 562 function createFileEncoder(path, schema, opts) { > 563 var encoder = new BlockEncoder(schema, opts); > 564 encoder.pipe(fs.createWriteStream(path, {defaultEncoding: 'binary'})); > 565 return encoder; > 566 } > {code} > In such a situation, the caller should wait for the underlying stream's > {{'finish'}} event rather than {{encoder}}'s. Because when the {{encoder}} > has finished to write and the caller catch its {{'finish'}} event, the > underlying stream perhaps doesn't finished to write into the disk yet (cf. > https://stackoverflow.com/questions/40864496/detecting-the-end-of-a-piped-stream-operation). > I think BlockEncoder should expose a reference to the underlying piped > stream, so that users can listen its event (typically {{'finish'}}). -- This message was sent by Atlassian Jira (v8.3.4#803005)