[protobuf] Re: Issue 302 in protobuf: Skip(count) fails when count offset >2^32 count type should be off_t, not int
Comment #1 on issue 302 by fac2...@gmail.com: Skip(count) fails when count offset >2^32 count type should be off_t, not int http://code.google.com/p/protobuf/issues/detail?id=302 Here's a patch that fixes the issue. The patch was written by Kevin C. Dorff and myself and fixes the issue against protobuf 2.4.1. Please incorporate to the head branch. We distribute several tools that are affected by this issue and while we can distribute the patch, it would be easier if the official protobuf release had the fix. Also note that the Java code generated by protobuf supports takes a long as argument to the corresponding skip method, so the C++ version should really match. public long skip(long l) throws java.io.IOException Attachments: protobuf-2.4.1-icb-offset-patch.txt 10.5 KB -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 302 in protobuf: Skip(count) fails when count offset >2^32 count type should be off_t, not int
Updates: Status: WontFix Comment #2 on issue 302 by liuj...@google.com: Skip(count) fails when count offset >2^32 count type should be off_t, not int http://code.google.com/p/protobuf/issues/detail?id=302 Protobuf is not designed to handle large messages. We didn't consider messages larger than 2GB from the very beginning. Could you please try to break the files into smaller ones? If we really need to change the count type, we need to update all functions dealing with byte-size/offset, not only the Skip function. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 302 in protobuf: Skip(count) fails when count offset >2^32 count type should be off_t, not int
Comment #3 on issue 302 by fac2...@gmail.com: Skip(count) fails when count offset >2^32 count type should be off_t, not int http://code.google.com/p/protobuf/issues/detail?id=302 The issue is not about large messages, but about large files. We use collections of small messages stored in large files (see http://campagnelab.org/software/goby/tutorials/developing-with-goby/). Please reconsider the issue considering that: 1. the current C++ API is in effect incompatible with the Java API, that has always supported skipping 64 bits offsets in large files. 2. the issue is not about supporting large messages. 3. we have provided a patch that solves the problem (see post of June 6 2011). As can be shown in the patch, the issue is limited to changing a few signatures to prevent the compiler from rounding a 64 bit integer to a 32bit one. Thanks. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 302 in protobuf: Skip(count) fails when count offset >2^32 count type should be off_t, not int
Comment #4 on issue 302 by liuj...@google.com: Skip(count) fails when count offset >2^32 count type should be off_t, not int http://code.google.com/p/protobuf/issues/detail?id=302 Well, for the java version, I think you mean the LimitedInputStream. That class has to use long to override its super class java.io.FilterInputStream; If you take a look at the c.g.protobuf.CodedInputStream, it uses int32 for size type. public void skipRawBytes(final int size) throws IOException We cannot simply change the Skip() to accept int64 because: 1> In protobuf, we use "int" to describe buffer size, e.g. the constructor of CodedInputStream, PushLimit(), GetDirectBufferPointer(), etc. We cannot change Skip() to accept int64, while leaving others still expecting int32. Users may be confused by the inconsistent size type. Even if we change Skip() to int64, the rest of code will still assumes the underlying buffer size is within int32, which can cause potential problems. 2> ZeroCopyStream interface uses int32. Changing the interface function signature means we need to update all the implementation classes. We could do a large code refactoring inside google, but we cannot change code outside the company. Changing the interface may break existing opensource protobuf users. I believe there are many clients who implement ZeroCopyStream for their IO layer. For your case, I think there are many workarounds. e.g. ensuring the chunk doesn't exceed int32max, and use ZeroCopyStream to operate on each chunk. Or maybe implement your own ZeroCopyStream, which ports large int operation to protobuf API. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
