I have the need to read in Avro messages from files that inflate to sizes that are causing OOM errors due to the in memory representation of the inflated document exceeding 1.5GB of Heap. Is there a way to stream the file into the application, inflate it, and marshal the contents without pulling the entire message into memory or am I restricted to chunking only at the message level?
- Reading large Avro message in Java with minimal heap usage Richard Ney
