metesynnada commented on issue #3674: URL: https://github.com/apache/arrow-rs/issues/3674#issuecomment-1422339772
I intentionally make a test like this, to make it more clear. I think waiting for the next byte to produce a RecordBatch even if we have the necessary bytes is against the streaming semantics. We use this test pattern for testing stream pipelines. We test "If I gave a batch, can I get a batch as output?", but testing with actual files becomes "If I gave a batch_size + 1 row, can I get a batch as output?". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
