[ https://issues.apache.org/jira/browse/ARROW-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175753#comment-16175753 ]
Wes McKinney commented on ARROW-1589: ------------------------------------- Could you clarify what kinds of malformed input you are talking about? I am not sure it is a requirement for the stream reader to be able to consistently return errors on random bytes input. In Arrow we need to distinguish between "can't fail" and "can fail" errors. The "can't fail" errors you detect in debug builds with DCHECK assertions. These are the kinds of errors that can only occur if the library developer (for example, an Arrow Java developer or an Arrow C++ developer) has implemented something incorrectly. Unit tests or integration tests must be written to exercise relevant code paths to catch these issues. I have found the debug assertions are especially helpful when refactoring, and they cost nothing in release builds. In the case of reading record batches from a stream, i.e. according to the encapsulated message format described in http://arrow.apache.org/docs/ipc.html, if you are able to read the indicated number of metadata bytes from the stream, then it is assumed to be a valid Flatbuffer, and the sender has respected invariants that are detectable in an integration test -- we may check do some sanity checks of invariants such as the number of buffers in a record batch. Same goes for the message body. If a Flatbuffer is truly malformed in some way in a way that cannot be detected with debug assertions, I am unsure whether we can protect ourselves from segfaults. The sender of a record batch stream must be assumed to be trusted (i.e. you have adequate integration tests against it to catch "can't fail" exceptions) to proceed with reading a stream at all. > Fuzzing for certain input formats > --------------------------------- > > Key: ARROW-1589 > URL: https://issues.apache.org/jira/browse/ARROW-1589 > Project: Apache Arrow > Issue Type: Test > Reporter: Marco Neumann > Assignee: Marco Neumann > > The arrow lib should have fuzzing tests for certain input formats, e.g. for > reading record batches from streams. Ideally, malformed input must not crash > the system but must report a proper error. This could easily be implemented > e.g. w/ [libfuzzer|https://llvm.org/docs/LibFuzzer.html] in combination with > address sanitizer (that's already implemented by Arrow's build system). -- This message was sent by Atlassian JIRA (v6.4.14#64029)