[jira] [Updated] (AVRO-1081) GenericDatumWriter does not support native ByteBuffers
[ https://issues.apache.org/jira/browse/AVRO-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting updated AVRO-1081: --- Resolution: Fixed Status: Resolved (was: Patch Available) I committed this. > GenericDatumWriter does not support native ByteBuffers > -- > > Key: AVRO-1081 > URL: https://issues.apache.org/jira/browse/AVRO-1081 > Project: Avro > Issue Type: Bug >Affects Versions: 1.6.3 >Reporter: Robert Fuller >Assignee: Doug Cutting > Fix For: 1.7.0 > > Attachments: AVRO-1081.patch, ByteBufferTest.java, > ByteBufferTest.java, patch.diff.txt, patch.diff.txt > > > An exception is thrown when trying to encode bytes backed by a file. > java.lang.UnsupportedOperationException: null > at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968) ~[na:1.6.0_31] > at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61) > ~[avro-1.6.3.jar:1.6.3] > Note arrayOffset is an optional method, see: > http://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html#arrayOffset%28%29 > FileChannel returns native ByteBuffer not HeapedByteBuffer > See here: > http://mail-archives.apache.org/mod_mbox/avro-user/201202.mbox/%3ccb57f421.6bfe2%25sc...@richrelevance.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1081) GenericDatumWriter does not support native ByteBuffers
[ https://issues.apache.org/jira/browse/AVRO-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting updated AVRO-1081: --- Fix Version/s: 1.7.0 Assignee: Doug Cutting Status: Patch Available (was: Open) > GenericDatumWriter does not support native ByteBuffers > -- > > Key: AVRO-1081 > URL: https://issues.apache.org/jira/browse/AVRO-1081 > Project: Avro > Issue Type: Bug >Affects Versions: 1.6.3 >Reporter: Robert Fuller >Assignee: Doug Cutting > Fix For: 1.7.0 > > Attachments: AVRO-1081.patch, ByteBufferTest.java, > ByteBufferTest.java, patch.diff.txt, patch.diff.txt > > > An exception is thrown when trying to encode bytes backed by a file. > java.lang.UnsupportedOperationException: null > at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968) ~[na:1.6.0_31] > at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61) > ~[avro-1.6.3.jar:1.6.3] > Note arrayOffset is an optional method, see: > http://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html#arrayOffset%28%29 > FileChannel returns native ByteBuffer not HeapedByteBuffer > See here: > http://mail-archives.apache.org/mod_mbox/avro-user/201202.mbox/%3ccb57f421.6bfe2%25sc...@richrelevance.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1081) GenericDatumWriter does not support native ByteBuffers
[ https://issues.apache.org/jira/browse/AVRO-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting updated AVRO-1081: --- Attachment: AVRO-1081.patch Here's an updated version of the patch. Instead of changing ReflectData.getField() I changed ReflectDatumReader.readBytes(). Does this work for you? > GenericDatumWriter does not support native ByteBuffers > -- > > Key: AVRO-1081 > URL: https://issues.apache.org/jira/browse/AVRO-1081 > Project: Avro > Issue Type: Bug >Affects Versions: 1.6.3 >Reporter: Robert Fuller > Attachments: AVRO-1081.patch, ByteBufferTest.java, > ByteBufferTest.java, patch.diff.txt, patch.diff.txt > > > An exception is thrown when trying to encode bytes backed by a file. > java.lang.UnsupportedOperationException: null > at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968) ~[na:1.6.0_31] > at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61) > ~[avro-1.6.3.jar:1.6.3] > Note arrayOffset is an optional method, see: > http://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html#arrayOffset%28%29 > FileChannel returns native ByteBuffer not HeapedByteBuffer > See here: > http://mail-archives.apache.org/mod_mbox/avro-user/201202.mbox/%3ccb57f421.6bfe2%25sc...@richrelevance.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1081) GenericDatumWriter does not support native ByteBuffers
[ https://issues.apache.org/jira/browse/AVRO-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Fuller updated AVRO-1081: Attachment: patch.diff.txt ByteBufferTest.java Attached one possible way of fixing this. I am not 100% happy with the solution, but it should work for our case for now. We are writing several avro files concurrently from within a heavily multithreaded application, and cannot afford to load many of the files into memory at once at that point. When reading the files again (in hadoop m/r job) it's ok for now to read the bytes back to memory (will read them into a byte array, but for the purpose of the test wanted to put them into a byte buffer). Am also now wondering whether you might consider direct support for "File" (or something like FileData) in avro. This would allow a way to include into avro items which exceed the amount of available memory. But maybe it's not a requirement anybody else would have... > GenericDatumWriter does not support native ByteBuffers > -- > > Key: AVRO-1081 > URL: https://issues.apache.org/jira/browse/AVRO-1081 > Project: Avro > Issue Type: Bug >Affects Versions: 1.6.3 >Reporter: Robert Fuller > Attachments: ByteBufferTest.java, ByteBufferTest.java, > patch.diff.txt, patch.diff.txt > > > An exception is thrown when trying to encode bytes backed by a file. > java.lang.UnsupportedOperationException: null > at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968) ~[na:1.6.0_31] > at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61) > ~[avro-1.6.3.jar:1.6.3] > Note arrayOffset is an optional method, see: > http://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html#arrayOffset%28%29 > FileChannel returns native ByteBuffer not HeapedByteBuffer > See here: > http://mail-archives.apache.org/mod_mbox/avro-user/201202.mbox/%3ccb57f421.6bfe2%25sc...@richrelevance.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1081) GenericDatumWriter does not support native ByteBuffers
[ https://issues.apache.org/jira/browse/AVRO-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Fuller updated AVRO-1081: Attachment: patch.diff.txt ByteBufferTest.java Attached one possible way of fixing this. I am not 100% happy with the solution, but it should work for our case for now. We are writing several avro files concurrently from within a heavily multithreaded application, and cannot afford to load many of the files into memory at once at that point. When reading the files again (in hadoop m/r job) it's ok for now to read the bytes back to memory (will read them into a byte array, but for the purpose of the test wanted to put them into a byte buffer). Am also now wondering whether you might consider direct support for "File" (or something like FileData) in avro. This would allow a way to include into avro items which exceed the amount of available memory. But maybe it's not a requirement anybody else would have... > GenericDatumWriter does not support native ByteBuffers > -- > > Key: AVRO-1081 > URL: https://issues.apache.org/jira/browse/AVRO-1081 > Project: Avro > Issue Type: Bug >Affects Versions: 1.6.3 >Reporter: Robert Fuller > Attachments: ByteBufferTest.java, ByteBufferTest.java, > patch.diff.txt, patch.diff.txt > > > An exception is thrown when trying to encode bytes backed by a file. > java.lang.UnsupportedOperationException: null > at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968) ~[na:1.6.0_31] > at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61) > ~[avro-1.6.3.jar:1.6.3] > Note arrayOffset is an optional method, see: > http://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html#arrayOffset%28%29 > FileChannel returns native ByteBuffer not HeapedByteBuffer > See here: > http://mail-archives.apache.org/mod_mbox/avro-user/201202.mbox/%3ccb57f421.6bfe2%25sc...@richrelevance.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
