[
https://issues.apache.org/jira/browse/AVRO-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Fuller updated AVRO-1081:
--------------------------------
Attachment: patch.diff.txt
ByteBufferTest.java
Attached one possible way of fixing this.
I am not 100% happy with the solution, but it should work for our case for now.
We are writing several avro files concurrently from within a heavily
multithreaded application, and cannot afford to load many of the files into
memory at once at that point.
When reading the files again (in hadoop m/r job) it's ok for now to read the
bytes back to memory (will read them into a byte array, but for the purpose of
the test wanted to put them into a byte buffer).
Am also now wondering whether you might consider direct support for "File" (or
something like FileData) in avro. This would allow a way to include into avro
items which exceed the amount of available memory. But maybe it's not a
requirement anybody else would have...
> GenericDatumWriter does not support native ByteBuffers
> ------------------------------------------------------
>
> Key: AVRO-1081
> URL: https://issues.apache.org/jira/browse/AVRO-1081
> Project: Avro
> Issue Type: Bug
> Affects Versions: 1.6.3
> Reporter: Robert Fuller
> Attachments: ByteBufferTest.java, ByteBufferTest.java,
> patch.diff.txt, patch.diff.txt
>
>
> An exception is thrown when trying to encode bytes backed by a file.
> java.lang.UnsupportedOperationException: null
> at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968) ~[na:1.6.0_31]
> at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61)
> ~[avro-1.6.3.jar:1.6.3]
> Note arrayOffset is an optional method, see:
> http://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html#arrayOffset%28%29
> FileChannel returns native ByteBuffer not HeapedByteBuffer
> See here:
> http://mail-archives.apache.org/mod_mbox/avro-user/201202.mbox/%3ccb57f421.6bfe2%25sc...@richrelevance.com%3E
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
